Lecture Notes

Discrete Structures II (COMP 2805)

Anil Maheshwari Michiel Smid

School of Computer Science

Carleton University

E-mail: {anil,michiel}@scs.carleton.ca

November 16, 2005

ii

Contents

Preface iii 1 Introduction 1 2 Finite Automata and Regular Languages 23 3 Context-Free Languages 77 4 Turing Machines and the Church-Turing Thesis 117 5 Decidable and Undecidable Languages 135 6 Complexity Theory 157 7 Summary 199 iv CONTENTS

Preface

This is a collection of notes for the course Discrete Structures II (COMP 2805), taught at Carleton University.

Please let us know if you find errors, typos, simpler proofs, comments, omissions, or if you think that some parts of the notes “need improvement”.

Besides reading these notes, we recommend that you also take a look at one or more of the following reference books:

• Introduction to the Theory of Computation, by Michael Sipser, PWS Publishing Company, Boston, 1997.

• Elements of the Theory of Computation (second edition), by Harry Lewis and Christos Papadimitriou, Prentice-Hall, 1998.

• Introduction to Languages and the Theory of Computation (third edition), by John Martin, McGraw-Hill, 2003.

• Introduction to Automata Theory, Languages, and Computation (second edition), by John Hopcroft, Rajeev Motwani, Jeffrey Ullman, Addison Wesley, 2001. vi CONTENTS

Chapter 1

Introduction

1.1 Purpose and motivation

This course is on the Theory of Computation, which tries to answers the following questions:

• What are the mathematical properties of computer hardware and software?

• What is a computation, and what is an algorithm? Can we give rigorous mathematical definitions of these notions?

• What are the limitations of computers? Can “everything” be computed? (As we will see, the answer to this question is “no”.)

Central Question in the Theory of Computation: What are the fundamental capabilities and limitations of computers?

This question was asked by mathematicians in the 1930’s, when they were trying to understand the meaning of a “computation”. The question showed up, because they wanted to know whether all mathematical problems can be solved in a systematic way. The research that started in those days led to computers as we know them today.

Nowadays, the Theory of Computation can be divided into the following three areas: Complexity Theory, Computability Theory, and Automata Theory.

1.1.1 Complexity theory

The main question asked in this area is “What makes some problems computationally hard and other problems easy?”

Informally, a problem is called “easy”, if it is efficiently solvable. Examples of “easy” problems are (i) sorting a sequence of, say, 1,000,000 numbers, (ii) searching for a name in a telephone directory, and (iii) computing the fastest way to drive from Ottawa to Miami. On the other hand, a problem is called “hard”, if it cannot be solved efficiently, or if we don’t know whether it can be solved efficiently. Examples of “hard” problems are (i) time table scheduling for all courses at Carleton, (ii) factoring a 300-digit integer into its prime factors, and (iii) computing a layout for chips in VLSI.

Central Question in Complexity Theory: Classify problems according to their degree of “difficulty”. Give a rigorous proof that problems that seem to be “hard” are really “hard”.

The major breakthrough in this area came in 1969 when Stefan Cook was able to separate the problems that can be efficiently solved by computers from those that computers will take enormous time to solve except for very small instances of them. He formally defined the notion of intractability and posed the notorious P vs N P problem, one of the seven Millennium problems (see http://www.claymath.org/millennium/).

1.1.2 Computability theory

In the 1930’s, Gödel, Turing, and Church discovered that some of the fundamental mathematical problems cannot be solved by “computer”. (This may sound strange, because computers were invented only in the 1940’s). An example of such a problem is “Is an arbitrary mathematical statement true or false?” To attack such a problem, we need formal definitions of the notions of

• computer,

• algorithm, and

• computation.

The theoretical models that were proposed in order to understand solvable and unsolvable problems led to the development of real computers.

Central Question in Computability Theory: Classify problems as being solvable or unsolvable.

1.1.3 Automata theory

Automata Theory is the study of abstract computing devices. It deals with definitions and properties of different types of “computation models”. Examples of such models are:

• Finite Automata. These are used in text processing, compilers, and hardware design.

• Context Free Grammars. These are used to define programming languages and in Artificial Intelligence.

• Turing Machines. These form a simple abstract model of a “real” computer, such as your PC at home.

Central Question in Automata Theory: Do these models have the same power, or can one model solve more problems than the other?

In 1930’s Turing studied the computing device, called Turing Machine, with the aim of mathematically defining the distinction between what that machine can do and what it cannot do (capabilities and limitations). It turns out that that theory is also applicable to present day computers! In the next two decades simpler machines than Turing Machines were proposed with the aim of modeling the brain. But these machines turned out to be very useful for many other purposes. These machines include Finite Automata, Context Free Grammars and Pushdown Automata.

1.1.4 This course

In this course, we will study these three topics in reverse order: We will start with Automata Theory, followed by Computability Theory, and finally, we will consider Complexity Theory.

Actually, before we start, we will review some mathematical proof techniques. As you may guess, this is a fairly theoretical course, with lots of definitions, theorems, and proofs. You may guess this course is fun stuff for Math lovers, but boring and irrelevant for others. You guessed it wrong, and here are the reasons:

1. This course is about the fundamental capabilities and limitations of computers. These topics form the core of computer science.

2. It is about Mathematical properties of computer hardware and software.

3. This theory is very much relevant to practice, for example, in the design of new programming languages, compilers, string searching, pattern matching, computer security, artificial intelligence, etc., etc.

4. This course helps you to learn problem solving skills. Theory teaches you how to think, prove, argue, solve problems, express, and abstract.

5. This theory simplifies the complex computers to an abstract and simple mathematical model, and helps you to understand them better.

6. This course is about rigorously analyzing capabilities and limitations of systems.

Where does this course fit in the Computer Science Curriculum at Carleton University? It is a theory course that is the second part in the series COMP 1805, COMP 2805, COMP 3804, and COMP 4804. This course also widens your understanding of computers and will influence other courses including Compilers, Design of Programming Languages, and Artificial Intelligence.

1.2 Mathematical preliminaries

Throughout this course, we will assume that you know the following mathematical concepts:

1. A set is a collection of well-defined objects. Examples are (i) the set of all Dutch Olympic Gold Medallists, (ii) the set of all parks in Ottawa, and (iii) the set of all even natural numbers.

2. The set of natural numbers is ℕ = {1, 2, 3, …}.

3. The set of integers is ℤ = {…, −3, −2, −1, 0, 1, 2, 3, …}.

4. The set of rational numbers is ℚ = {m/n : m ∈ ℤ, n ∈ ℤ, n ≠ 0}.

5. The set of real numbers is denoted by ℝ.

6. If A and B are sets, then A is a subset of B, written as A ⊆ B, if every element of A is also an element of B. For example, the set of even natural numbers is a subset of the set of all natural numbers. Every set A is a subset of itself, i.e., A ⊆ A. The empty set is a subset of every set A, i.e., ∅ ⊆ A.

7. If A and B are two sets, then

(a) their union is defined as

A ∪ B = {x : x ∈ A or x ∈ B},

(b) their intersection is defined as

A ∩ B = {x : x ∈ A and x ∈ B},

(c) the Cartesian product of A and B is defined as

A × B = {(x, y) : x ∈ A and y ∈ B},

(d) the complement of A is defined as

\overline{A} = \{x : x ∉ A\}.

8. A binary relation on two sets A and B is a subset of A × B.

9. A function f from A to B, denoted by f : A → B, is a binary relation R, having the property that for each element a ∈ A, there is exactly one ordered pair in R, whose first component is a. We will also say that f(a) = b, or f maps a to b, or the image of a under f is b. The set A is called the domain of f, and the set

{b ∈ B : there is an a ∈ A with f(a) = b}

is called the range of f.

10. A function f : A → B is one-to-one (or injective), if for any two distinct elements a and a′ in A, we have f(a) ≠ f(a′). The function f is onto (or surjective), if for each element b ∈ B, there exists an element a ∈ A, such that f(a) = b; in other words, the range of f is equal to the set B. A function f is a bijection, if f is both injective and surjective.

11. A binary relation R ⊆ A × A is an equivalence relation, if it satisfies the following three conditions:

(a) R is reflexive: For every element in a ∈ A, we have (a, a) ∈ R.

(b) R is symmetric: For all a and b in A, if (a, b) ∈ R, then also (b, a) ∈ R.

(c) R is transitive: For all a, b, and c in A, if (a, b) ∈ R and (b, c) ∈ R, then also (a, c) ∈ R.

12. A graph G = (V, E) is a pair consisting of a set V, whose elements are called vertices, and a set E, where each element of E is a pair of distinct vertices. The elements of E are called edges. The figure below shows some well-known graphs: K_{5} (the complete graph on five vertices), K_{3,3} (the complete bipartite graph on 2 × 3 = 6 vertices), and the Peterson graph.

K_{5}  K_{3,3}  Peterson graph

The degree of a vertex v, denoted by deg(v), is defined to be the number of edges that are incident on v.

A path in a graph is a sequence of vertices that are connected by edges. A path is a cycle, if it starts and ends at the same vertex. A simple path is a path without any repeated vertices. A graph is connected, if there is a path between every pair of vertices.

13. If each edge in the graph is oriented than the graph is called as a directed graph.

14. The Boolean values are 1 and 0, that represent true and false, respectively. The basic Boolean operations include

(a) negation (or NOT ), represented by ¬,

(b) conjunction (or AND), represented by ∧,

(c) disjunction (or OR), represented by ∨,

(d) exclusive-or (or XOR), represented by ⊕,

(e) equivalence, represented by ↔ or ⇔,

(f) implication, represented by → or ⇒.

The following table explains the meanings of these operations.

NOT  AND  OR  XOR equivalence implication
¬0 = 1 0 ∧ 0 = 0 0 ∨ 0 = 0 0 ⊕ 0 = 0 0 ↔ 0 = 1 0 → 0 = 1
¬1 = 0 0 ∧ 1 = 0 0 ∨ 1 = 1 0 ⊕ 1 = 1 0 ↔ 1 = 0 0 → 1 = 1
1 ∧ 0 = 0 1 ∨ 0 = 1 1 ⊕ 0 = 1 1 ↔ 0 = 0 1 → 0 = 0
1 ∧ 1 = 1 1 ∨ 1 = 1 1 ⊕ 1 = 0 1 ↔ 1 = 1 1 → 1 = 1

15. In the context of strings, an alphabet is a finite set, whose elements are called symbols. Examples of alphabets are Σ = {0, 1} and Σ = {a, b, c, …, z}.

16. A string over an alphabet Σ is a finite sequence of symbols, where each symbol is an element of Σ. The length of a string w, denoted by |w|, is the number of symbols contained in w. The empty string, denoted by ϵ, is the string having length zero. For example, if the alphabet Σ is equal to {0, 1}, then 10, 1000, 0, 101, and ϵ are strings over Σ, having lengths 2, 4, 1, 3, and 0, respectively. The strings over the binary alphabet {0, 1} are called binary strings.

17. The concatenation of two strings x and y is the string xy, that is the string x followed by the string y. The reversal of a string x = x_{1}x_{2} ⋯ x_{n}, denoted by x^{R}, is the string x^{R} = x_{n}x_{n−1} ⋯ x_{2}x_{1}. If x = ϵ then x^{R} = ϵ.

18. A language is a set of strings.

1.3 Proof techniques

In mathematics, a theorem is a statement that is true. A proof is a sequence of mathematical statements that form an argument to show that a theorem is true. The statements in the proof of a theorem include axioms (assumptions about the underlying mathematical structures), hypotheses of the theorem to be proved, and previously proved theorems. The main question is “How do we go about proving theorems?” This question is similar to the question of how to solve a given problem. Of course, the answer is that finding proofs, or solving problems, is not easy; otherwise life would be dull! There is no specified way of coming up with a proof, but there are some generic strategies that could be of help. In this section, we review some of these strategies, that will be sufficient for this course. The best way to get a feeling of how to come up with a proof is by solving a large number of problems. Here are some useful tips. (You may take a look at the book How to Solve It, by G. Pólya).

1. Read and completely understand the statement of the theorem to be proved. Most often this is the hardest part.

2. Sometimes, theorems contain theorems inside them. For example, “Property A if and only if property B”, requires showing two statements:

(a) If property A is true, then property B is true (A ⇒ B).

(b) If property B is true, then property A is true (B ⇒ A).

Another example is the theorem “Set A equals set B.” To prove this, we need to prove that A ⊆ B and B ⊆ A. That is, we need to show that each element of set A is in set B, and that each element of set B is in set A.

3. Try to work out a few simple cases of the theorem just to get the grip on it (i.e., crack a few simple cases first).

4. Try to write down the proof once you have it. This is to ensure the correctness of your proof. Often, mistakes are found at the time of writing.

5. Finding proofs take time, we do not come prewired to produce proofs. Be patient, think, express and write clearly and try to be precise as much as possible.

In the next sections, we will go through some of the proof strategies.

1.3.1 Direct proofs

As the name suggests, in a direct proof of a theorem, we just approach the theorem directly.

Theorem 1.3.1 If n is an odd positive integer, then n^{2} is odd as well.

Proof. An odd positive integer n can be written as n = 2k + 1, for some integer k ≥ 0. Then

n^{2} = (2k + 1)^{2} = 4k^{2} + 4k + 1 = 2(2k^{2} + 2k) + 1.

Since 2(2k^{2} + 2k) is even, and one more than even is odd, we can conclude that n^{2} is odd.

Theorem 1.3.2 Let G = (V, E) be a graph. Then the sum of the degrees of all vertices is an even integer, i.e.,

∑_{v∈V} deg(v)

is even.

Proof. If you don’t see the meaning of this statement, then first try it out for a few graphs. The reason why the statement holds is very simple: each edge contributes 2 to the summation (because an edge is incident on exactly two distinct vertices).

Actually, the proof above proves the following theorem.

Theorem 1.3.3 Let G = (V, E) be a graph. Then the sum of the degrees of all vertices is equal to twice the number of edges, i.e.,

∑_{v∈V} deg(v) = 2|E|.

1.3.2 Constructive proofs

This technique not only shows the existence of a certain object, it actually gives a method of creating it. Here is how a constructive proof looks like:

Theorem 1.3.4 There exists an object with property P.

Proof. Here is the object: […] And here is the proof that the object satisfies property P: […]

Here is an example of constructive proof. A graph is called 3-regular, if each vertex has degree three.

Theorem 1.3.5 For every even integer n ≥ 4, there exists a 3-regular graph with n vertices.

Proof. Define

V = {0, 1, 2, …, n − 1},

and

E = {{i, i+1} : 0 ≤ i ≤ n−2}∪{{n−1, 0}}∪{{i, i+n/2} : 0 ≤ i ≤ n/2−1}.

Then the graph G = (V, E) is 3-regular.

Convince yourself that this graph is indeed 3-regular. It may help to draw the graph for, say n = 8.

1.3.3 Proofs by contradiction

This is how a proof by contradiction looks like:

Theorem 1.3.6 Statement s is true.

Proof. Assume that statement s is false. Then, derive a contradiction (such as 1 + 1 = 3).

In other words, show that ¬s ⇒ false. This is sufficient, because the contrapositive of ¬s ⇒ false is true ⇒ s. The latter logical formula is equivalent to s, and that is what we wanted to show.

Below, we give two examples of proofs by contradiction.

Theorem 1.3.7 Let n be a positive integer. If n^{2} is even, then n is even.

Proof. We will prove the theorem by contradiction. So we assume that n^{2} is even, but n is odd. Since n is odd, we know from Theorem 1.3.1 that n^{2} is odd. This is a contradiction, because we assumed that n^{2} is even.

Theorem 1.3.8 \sqrt{2} is irrational, i.e., \sqrt{2} cannot be written as a fraction of two integers m and n.

Proof. We will prove the theorem by contradiction. So we assume that \sqrt{2} is rational. Then \sqrt{2} can be written as a fraction of two integers, \sqrt{2} = m/n, where m ≥ 1 and n ≥ 1. We may assume that m and n are not both even, because otherwise we can get rid of the common factors. By squaring \sqrt{2} = m/n, we get 2n^{2} = m^{2}. This implies that m^{2} is even. Then, by Theorem 1.3.7, m is even, which means that we can write m as m = 2k, for some positive integer k. It follows that 2n^{2} = m^{2} = 4k^{2}, which implies that n^{2} = 2k^{2}. Hence, n^{2} is even. Again by theorem 1.3.7, it follows that n is even.

We have shown that m and n are both even. But we know that m and n are not both even. Hence, we have a contradiction. Our assumption that \sqrt{2} is rational is wrong. Thus, we can conclude that \sqrt{2} is irrational.

There is a nice discussion of this proof in the book My Brain is Open: The Mathematical Journeys of Paul Erdős by B. Schechter.

1.3.4 Countable sets and Cantor’s Diagonalization

Let A and B be two sets, and let f : A → B be a function. Recall that f is called a bijection, if

• f is one-to-one (or injective), i.e., for any two distinct elements a and a′ in A, we have f(a) ≠ f(a′), and

• f is onto (or surjective), i.e., for each element b ∈ B, there exists an element a ∈ A, such that f(a) = b.

The set of natural numbers is denoted by ℕ. That is, ℕ = {1, 2, 3, …}.

Definition 1.3.9 Let A and B be two sets. We say that A and B have the same size, if there exists a bijection f : A → B.

Definition 1.3.10 Let A be a set. We say that A is countable, if A is finite, or A and ℕ have the same size.

In other words, if A is an infinite and countable set, then there exists a bijection f : ℕ → A, and we can write A as

A = {f(1), f(2), f(3), f(4), …}.

Since f is a bijection, every element of A occurs exactly once in the set on the right-hand side. This means that we can number the elements of A using the positive integers: Every element of A receives a unique number.

Theorem 1.3.11 The following sets are countable:

1. The set ℤ of integers:

ℤ = {…, −3, −2, −1, 0, 1, 2, 3, …}.

2. The Cartesian product ℕ × ℕ:

ℕ × ℕ = {(m, n) : m ∈ ℕ, n ∈ ℕ}.

3. The set ℚ of rational numbers:

ℚ = {m/n : m ∈ ℤ, n ∈ ℤ, n ≠ 0}.

Proof. To prove that the set ℤ is countable, we have to give each element of ℤ a unique number in ℕ. We obtain this numbering, by listing the elements of ℤ in the following order:

0, 1, −1, 2, −2, 3, −3, 4, −4, …

In this (infinite) list, every element of ℤ occurs exactly once. The number of an element of ℤ is given by its position in this list.

Formally, define the function f : ℕ → ℤ by

f(n) = \{ \begin{matrix}n/2 & if n is even, \\ −(n − 1)/2 & if n is odd. \end{matrix}

This function f is a bijection and, therefore, the sets ℕ and ℤ have the same size. Hence, the set ℤ is countable.

For the proofs of the other two claims, we refer to the course COMP 1805.

Theorem 1.3.12 The set ℝ of real numbers is not countable.

Proof. Define

A = {x ∈ ℝ : 0 ≤ x < 1}.

We will prove that the set A is not countable. This will imply that the set ℝ is not countable, because A ⊆ ℝ.

The proof that A is not countable is by contradiction. So we assume that A is countable. Then there exists a bijection f : ℕ → A. Hence, we can write

A = {f(1), f(2), f(3), …},  (1.1)

where every element of A occurs exactly once in the set on the right-hand side. Hence, for each n ∈ ℕ, f(n) is a real number between zero and one.

Consider the real number f(1). We can write this number in decimal notation as

f(1) = 0.d_{11}d_{12}d_{13} …,

where each d_{1i} is a digit in the set {0, 1, 2, …, 9}. In general, for every n ∈ ℕ, we can write the real number f(n) as

f(n) = 0.d_{n1}d_{n2}d_{n3} …,

where, again, each d_{ni} is a digit in {0, 1, 2, …, 9}.

We define the real number

x = 0.d_{1}d_{2}d_{3} …,

where, for each n ≥ 1,

d_{n} = \{ \begin{matrix}4 & if d_{nn} ≠ 4, \\ 5 & if d_{nn} = 4. \end{matrix}

Observe that x is a real number between zero and one, i.e., x ∈ A. Therefore, by (5.1), there is an element n ∈ ℕ, such that f(n) = x. We compare the n-th digits of f(n) and x:

• The n-th digit of f(n) is equal to d_{nn}.

• The n-th digit of x is equal to d_{n}.

Since f(n) and x are equal, their n-th digits must be equal, i.e., d_{nn} = d_{n}. But, by the definition of d_{n}, we have d_{nn} ≠ d_{n}. This is a contradiction and, therefore, the set A is not countable.

The above theorem is due to Georg Cantor (1879) and this proof method is called as the Cantor’s diagonalization method. This is used extensively in Mathematical Logic as well as in the later part of this course.

1.3.5 The pigeon hole principle

This is a simple principle with surprising consequences. It states that if n + 1 or more objects are placed into n boxes, then there is at least one box containing two or more objects. In other words, if A and B are two sets such that |A| > |B|, then there is no one-to-one function from A to B.

Theorem 1.3.13 Let n be a positive integer. Every sequence of n^{2} + 1 distinct real numbers contains a subsequence of length n + 1 that is either increasing or decreasing.

Proof. For example consider the sequence (20, 10, 9, 7, 11, 2, 21, 1, 20, 31) of 10 = 3^{2} + 1 numbers. This sequence contains an increasing subsequence of length 4 = 3 + 1, namely (10, 11, 21, 31).

The proof of this theorem is by contradiction, and uses the pigeon hole principle.

Let (a_{1}, a_{2}, …, a_{n^{2}+1}) be an arbitrary sequence of n^{2} + 1 distinct real numbers. For each i with 1 ≤ i ≤ n^{2} + 1, let inc_{i} denote the length of the longest increasing subsequence that starts at a_{i}, and let dec_{i} denote the length of the longest decreasing subsequence that starts at a_{i}.

Using this notation, the claim in the theorem can be formulated as follows: There is an index i such that inc_{i} ≥ n + 1 or dec_{i} ≥ n + 1.

We will prove the claim by contradiction. So we assume that inc_{i} ≤ n and dec_{i} ≤ n for all i with 1 ≤ i ≤ n^{2} + 1.

Consider the set

B = {(b, c) : 1 ≤ b ≤ n, 1 ≤ c ≤ n},

and think of the elements of B as being boxes. For each i with 1 ≤ i ≤ n^{2}+1, the pair (inc_{i}, dec_{i}) is an element of B. So we have n^{2}+1 elements (inc_{i}, dec_{i}), which are placed in the n^{2} boxes of B. By the pigeon hole principle, there must be a box that contains two (or more) elements. In other words, there exist two integers i and j such that i < j and

(inc_{i}, dec_{i}) = (inc_{j}, dec_{j}).

Recall that the elements in the sequence are distinct. Hence, a_{i} ≠ a_{j}. We consider two cases.

First assume that a_{i} < a_{j}. Then the length of the longest increasing subsequence starting at a_{i} must be at least 1+inc_{j}, because we can append a_{i} to the longest increasing subsequence starting at a_{j}. Therefore, inc_{i} ≠ inc_{j}, which is a contradiction.

The second case is when a_{i} > a_{j}. Then the length of the longest decreasing subsequence starting at a_{i} must be at least 1+dec_{j}, because we can append a_{i} to the longest decreasing subsequence starting at a_{j}. Therefore, dec_{i} ≠ dec_{j}, which is again a contradiction.

1.3.6 Proofs by induction

This is a very powerful and important technique for proving theorems (especially in computer science).

For each positive integer n, let P (n) be a mathematical statement that depends on n. Assume we wish to prove that P (n) is true for all positive integers n. A proof by induction of such a statement is carried out as follows:

Basis: Prove that P (1) is true.

Induction step: Prove that for all n ≥ 1, the following holds: If P (n) is true, then P (n + 1) is also true.

In the induction step, we choose an arbitrary integer n ≥ 1, and assume that P (n) is true; this is called the induction hypothesis. Then we prove that P (n + 1) is also true.

Theorem 1.3.14 For all positive integers n, we have

1 + 2 + 3 + … + n = \frac{n(n + 1)}{2}.

Proof. We start with the basis of the induction. If n = 1, then the left-hand side is equal to 1, and so is the right-hand side. So the theorem is true for n = 1.

For the induction step, let n ≥ 1, and assume that the theorem is true for n, i.e., assume that

1 + 2 + 3 + … + n = \frac{n(n + 1)}{2}.

We have to prove that the theorem is true for n + 1, i.e., we have to prove that

1 + 2 + 3 + … + (n + 1) = \frac{(n + 1)(n + 2)}{2}.

Here is the proof:

1 + 2 + 3 + … + (n + 1) = \underbrace{1 + 2 + 3 + … + n}_{= \frac{n(n+1)}{2}} +(n + 1)

= \frac{n(n + 1)}{2} + (n + 1)
= \frac{(n + 1)(n + 2)}{2}.

By the way, here is an alternative proof of the theorem above: Let S = 1 + 2 + 3 + … + n. Then,

\begin{aligned}S & = & 1 & + & 2 & + & 3 & + & … & + & (n − 2) & + & (n − 1) & + & n \\ S & = & n & + & (n − 1) & + & (n − 2) & + & … & + & 3 & + & 2 & + & 1 \\ 2S & = & (n + 1) & + & (n + 1) & + & (n + 1) & + & … & + & (n + 1) & + & (n + 1) & + & (n + 1) \end{aligned}

Since there are n terms on the right-hand side, we have 2S = n(n + 1). but this means that S = n(n + 1)/2.

Theorem 1.3.15 For every positive integer n, a − b is a factor of a^{n} − b^{n}.

Proof. A direct proof can be given by providing a factorization of a^{n} − b^{n}:

a^{n} − b^{n} = (a − b)(a^{n−1} + a^{n−2}b + a^{n−3}b^{2} + … + ab^{n−2} + b^{n−1}).

We now prove the theorem by induction. For the basis, let n = 1. The claim in the theorem is “a − b is a factor of a − b”, which is obviously true.

Let n ≥ 1, and assume that a − b is a factor of a^{n} − b^{n}. We have to prove that a − b is a factor of a^{n+1} − b^{n+1}. We have

a^{n+1} − b^{n+1} = a^{n+1} − a^{n}b + a^{n}b − b^{n+1} = a^{n}(a − b) + (a^{n} − b^{n})b.

The first term on the right-hand side is divisible by (a−b). By the induction hypothesis, the second term on the right-hand side is divisible by (a − b) as well. Therefore, the entire right-hand side is divisible by (a − b). since the right-hand side is equal to a^{n+1} − b^{n+1}, it follows that a − b is a factor of a^{n+1} − b^{n+1}.

We now give an alternative proof of Theorem 1.3.3:

Theorem 1.3.16 Let G = (V, E) be a graph with m edges. Then the sum of the degrees of all vertices is equal to twice the number of edges, i.e.,

∑_{v∈V} deg(v) = 2m.

Proof. The proof is by induction on the number m of edges. For the basis of the induction, assume that m = 0. Then the graph G does not contain any edges and, therefore, ∑_{v∈V} deg(v) = 0. Thus, the theorem is true if m = 0.

Let m ≥ 0, and assume that the theorem is true for every graph with m edges. Let G be an arbitrary graph with m + 1 edges. We have to prove that ∑_{v∈V} deg(v) = 2(m + 1).

Let {a, b} be an arbitrary edge in G, and let G′ be the graph obtained from G by removing the edge {a, b}. Since G′ has m edges, we know from the induction hypothesis that the sum of the degrees of all vertices in G′ is equal to 2m. Using this, we obtain

∑_{v∈G} deg(v) = ∑_{v∈G′} deg(v) + 2 = 2m + 2 = 2(m + 1).

1.3.7 More examples of proofs

Recall Theorem 1.3.5, which states that for every even integer n ≥ 4, there exists a 3-regular graph with n vertices. The following theorem explains why we stated this theorem for even values of n.

Theorem 1.3.17 Let n ≥ 5 be an odd integer. There is no 3-regular graph with n vertices.

Proof. The proof is by contradiction. So we assume that there exists a graph G = (V, E) with n vertices that is 3-regular. Let m be the number of edges in G. Since deg(v) = 3 for every vertex, we have

∑_{v∈V} deg(v) = 3n.

On the other hand, by Theorem 1.3.3, we have

∑_{v∈V} deg(v) = 2m.

It follows that 3n = 2m, which can be rewritten as m = 3n/2. Since m is an integer, and since gcd(2, 3) = 1, n/2 must be an integer. Hence, n is even, which is a contradiction.

Let K_{n} be the complete graph on n vertices. This graph has a vertex set of size n, and every pair of distinct vertices is joined by an edge.

If G = (V, E) is a graph with n vertices, then the complement \overline{G} of G is the graph with vertex set V that consists of those edges of K_{n} that are not present in G.

Theorem 1.3.18 Let n ≥ 2, and let G be a graph on n vertices. Then G is connected or \overline{G} is connected.

Proof. We prove the theorem by induction on the number n of vertices. For the basis, assume that n = 2. There are two possibilities for the graph G:

1. G contains one edge. In this case, G is connected.

2. G does not contain an edge. In this case, the complement \overline{G} contains one edge and, therefore, \overline{G} is connected.

So for n = 2, the theorem is true.

Let n ≥ 2, and assume that the theorem is true for every graph with n vertices. Let G be graph with n + 1 vertices. We have to prove that G is connected or \overline{G} is connected. We consider three cases.

Case 1: There is a vertex v whose degree in G is equal to n.

Since G has n + 1 vertices, this means that v is connected by an edge to every other vertex of G. Therefore, G is connected.

Case 2: There is a vertex v whose degree in G is equal to 0.

In this case, the degree of v in the graph \overline{G} is equal to n. Since \overline{G} has n+1 vertices, this means that v is connected by an edge to every other vertex of \overline{G}. Therefore, \overline{G} is connected.

Case 3: For every vertex v, the degree of v in G is in {1, 2, …, n − 1}.

Let v be an arbitrary vertex of G. Let G′ be the graph obtained by deleting from G the vertex v, together with all edges that are incident on v. Since G′ has n vertices, we know from the induction hypothesis that G′ is connected or \overline{G′} is connected.

Let us first assume that G′ is connected. Then the graph G is connected as well, because there is at least one edge in G between v and some vertex of G′.

If G′ is not connected, then \overline{G′} must be connected. Since we are in Case 3, we know that the degree of v in G is in the set {1, 2, …, n − 1}. It follows that the degree of v in the graph \overline{G} is in this set as well. Hence, there is at least one edge in \overline{G} between v and some vertex in \overline{G′}. This implies that \overline{G} is connected.

A graph is said to be planar, if it can be drawn (a better term is “embedded”) in the plane in such a way that no two edges intersect, except possibly at their endpoints. An embedding of a planar graph consists of vertices, edges, and faces. In the example below, there are 11 vertices, 18 edges, and 9 faces (including the unbounded face).

The following theorem is known as Euler’s theorem for planar graphs. Apparently, this theorem was discovered by Euler around 1750. Legendre gave the first proof in 1794, see

       http://www.ics.uci.edu/~eppstein/junkyard/euler/

Theorem 1.3.19 (Euler) Consider an embedding of a planar graph G. Let v, e, and f be the number of vertices, edges, and faces (including the single unbounded face) of this embedding, respectively. Moreover, let c be the number of connected components of G. Then

v − e + f = c + 1.

Proof. The proof is by induction on the number of edges of G. To be more precise, we start with graph having no edges, and prove that the theorem holds for this case. Then, we add the edges one by one, and show that the relation v − e + f = c + 1 is maintained.

So we first assume that G has no edges, i.e., e = 0. Then the embedding consists of a collection of v points. In this case, we have f = 1 and c = v. Hence, the relation v − e + f = c + 1 holds.

Let e > 0 and assume that Euler’s formula holds for a subgraph of G having e − 1 edges. Let {u, v} be an edge of G that is not in the subgraph, and add this edge to the subgraph. There are two cases depending on whether this new edge joins two connected components or joins two vertices in the same connected component.

Case 1: The new edge {u, v} joins two connected components.

In this case, the number of vertices and the number of faces do not change, the number of connected components goes down by 1, and the number of edges increases by 1. It follows that the relation in the theorem is still valid.

Case 2: The new edge {u, v} joins two vertices in the same connected component.

In this case, the number of vertices and the number of connected components do not change, the number of edges increases by 1, and the number of faces increases by 1 (because the new edge splits one face into two faces). Therefore, the relation in the theorem is still valid.

Euler’s theorem is usually stated as follows:

Theorem 1.3.20 (Euler) Consider an embedding of a connected planar graph G. Let v, e, and f be the number of vertices, edges, and faces (including the single unbounded face) of this embedding, respectively. Then

v − e + f = 2.

If you like surprising proofs of various mathematical results, you should read the book Proofs from THE BOOK by Aigner and Ziegler.

1.4 Exercises

1.1 Use induction to prove that every positive integer can be written as a product of prime numbers.

1.2 For every prime number p, prove that \sqrt{p} is irrational.

1.3 Let n be a positive integer that is not a perfect square. Prove that \sqrt{n} is irrational.

1.4 Prove by induction that n^{4} − 4n^{2} is divisible by 3, for all integers n ≥ 1.

1.5 Prove that

∑_{i=1}^{n} \frac{1}{i^{2}} < 2 − 1/n,

for every integer n ≥ 2.

1.6 Prove that 9 divides n^{3} + (n + 1)^{3} + (n + 2)^{3}, for every integer n ≥ 0.

1.7 In any set of n + 1 numbers from {1, 2, …, 2n}, there are always two numbers that are consecutive.

1.8 In any set of n + 1 numbers from {1, 2, …, 2n}, there are always two numbers such that one divides the other.

1.9 Show that 2^{2n} − 1 is divisible by 3 for any positive integer n.

1.10 Show that for strings x and y, the reversal of the concatenation of x and y is the concatenation of the reversal of y and the reversal of x, i.e. (xy)^{R}=y^{R}x^{R}.

1.11 Show that union of two countable sets is countable.

1.12 Show that union of a countable number of countable sets is countable.

1.13 Show that the set of all bit strings is countable.

1.14 Show that the set of all computer programs written in Java is countable. A program is a string of symbols from a finite set of alphabets.

1.15 Show that the set of functions from positive integers to the set {0,1,2,3,4,5,6,7,8,9} is uncountable.

1.16 A function is said to be computable, if there is a computer program that can compute the function. Show that there are functions that are not computable.

1.17 Let n ≥ 2, and consider the complete graph K_{n} on n vertices. Color each edge of this graph as either red or blue. Let R be the graph consisting of all the red edges, and let B be the graph consisting of all the blue edges. Show that R is connected or B is connected.

Finite Automata and Regular Languages

In this chapter, we introduce and analyze the class of languages that are known as regular languages. Informally, these languages can be “processed” by computers having a very small amount of memory.

2.1 An example: designing a toll gate

Before we give a formal definition of a finite automaton, we consider an example in which such an automaton shows up in a natural way. We consider the problem of designing a “computer” that controls a toll gate.

When a car driver arrives at the toll gate, the gate is closed. The gate opens as soon as the driver has payed 25 cents. We assume that we have only three coin denominations: 5, 10, and 25 cents. We also assume that no excess change is refunded.

After having arrived at the toll gate, the driver inserts a sequence of coins into the machine. At any moment, the machine has to decide whether or not to open the gate, i.e., whether or not the driver has paid 25 cents (or more). In order to decide this, the machine is in one of the following six states, at any moment during the process:

• The machine is in state q_{0}, if it has not collected any money yet.

• The machine is in state q_{1}, if it has collected exactly 5 cents.

• The machine is in state q_{2}, if it has collected exactly 10 cents.

24 CHAPTER 2. FINITE AUTOMATA AND REGULAR LANGUAGES

• The machine is in state q_{3}, if it has collected exactly 15 cents.

• The machine is in state q_{4}, if it has collected exactly 20 cents.

• The machine is in state q_{5}, if it has collected 25 cents or more.

Initially (when a driver arrives at the toll gate), the machine is in state q_{0}. Assume, for example, that the driver presents the sequence (10,5,5,10) of coins.

• After receiving the first 10 cents coin, the machine switches from state q_{0} to state q_{2}.

• After receiving the first 5 cents coin, the machine switches from state q_{2} to state q_{3}.

• After receiving the second 5 cents coin, the machine switches from state q_{3} to state q_{4}.

• After receiving the second 10 cents coin, the machine switches from state q_{4} to state q_{5}. At this moment, the gate opens. (Remember that no change is given.)

The figure below represents the behavior of the machine, for all possible sequences of coins.

5  5  5  5 10  10  10  10, 25 25 25 25 Start  5,10 25 5, 10, 25 q_{0}  q_{1}  q_{2}  q_{3}  q_{4}  q_{5}

Observe that the machine (or computer) only has to remember which state it is in at any given time. Thus, it needs only a very small amount of memory: It has to be able to distinguish between any one of six possible cases and, therefore, it needs a memory of only ⌈log 6⌉ = 3 bits.

2.2 Deterministic finite automata

Let us look at another example. Consider the following state diagram:

q_{1}  q_{2}  q_{3} 0  0 1  1  0, 1

We say that q_{1} is the start state, and q_{2} is an accept state. Consider the input string 1101. This string is processed in the following way:

• Initially, the machine is in the start state q_{1}.

• After having read the first 1, the machine switches from state q_{1} to state q_{2}.

• After having read the second 1, the machine switches from state q_{2} to state q_{2}. (So actually, it does not switch.)

• After having read the first 0, the machine switches from state q_{2} to state q_{3}.

• After having read the third 1, the machine switches from state q_{3} to state q_{2}.

After the entire string 1101 has been processed, the machine is in state q_{2}, which is an accept state. We say that the string 1101 is accepted by the machine.

When the machine gets the string 0101010 as input, it ends up in state q_{3} (after having read the entire string, and starting in the start state q_{1}). Since q_{3} is not an accept state, we say that the machine rejects the string 0101010.

We hope you are able to see that this machine accepts every binary string that ends with a 1. In fact, the machine accepts more strings:

• Every binary string having the property that there are an even number of 0s following the rightmost 1, is accepted by this machine.

Every other binary string is rejected by the machine.

Definition 2.2.1 A finite automaton is a 5-tuple M = (Q, Σ, δ, q, F) where 1. Q is a finite set, whose elements are called states,

2. Σ is a finite set, called the alphabet; the elements of Σ are called symbols,

3. δ : Q × Σ → Q is a function, called the transition function,

4. q is an element of Q; it is called the start state,

5. F is a subset of Q; the elements of F are called accept states.

You can think of the transition function δ as being the “program” of the finite automaton M = (Q, Σ, δ, q, F). This function tells us what M can do in “one step”:

• Let r be a state of Q, and let a be a symbol of the alphabet Σ. If the finite automaton M is in state r, and if it reads the symbol a, then it switches from state r to state δ(r, a).

The “computer” that we designed in the toll gate example is a finite automaton. For this example, we have Q = {q_{0}, q_{1}, q_{2}, q_{3}, q_{4}, q_{5}}, Σ = {5, 10, 25}, the start state is q_{0}, F = {q_{5}}, and δ is given by the following table:

5 10 25
q_{0} q_{1} q_{2} q_{5}
q_{1} q_{2} q_{3} q_{5}
q_{2} q_{3} q_{4} q_{5}
q_{3} q_{4} q_{5} q_{5}
q_{4} q_{5} q_{5} q_{5}
q_{5} q_{5} q_{5} q_{5}

The example given in the beginning of this section is also a finite automaton. For this example, we have Q = {q_{1}, q_{2}, q_{3}}, Σ = {0, 1}, the start state is q_{1}, F = {q_{2}}, and δ is given by the following table:

\begin{aligned}& 0 & 1 \\ q_{1} & q_{1} & q_{2} \\ q_{2} & q_{3} & q_{2} \\ q_{3} & q_{2} & q_{2} \end{aligned}

Let us denote this finite automaton by M. The language of M, denoted by L(M), is the set of all strings that are accepted by M. Then, for this example, we have

L(M) = {w : w contains at least one 1 and ends with an even number of 0s}.

We now give a formal definition of the language of a finite automaton:

Definition 2.2.2 Let M = (Q, Σ, δ, q, F) be a finite automaton, and let w = w_{1}w_{2} … w_{n} be a string over Σ. Define the sequence r_{0}, r_{1}, …, r_{n} of states, in the following way:

• r_{0} = q, and

• r_{i+1} = δ(r_{i}, w_{i+1}), for i = 0, 1, …, n − 1.

1. If r_{n} ∈ F, then we say that M accepts w.

2. If r_{n} ∉ F, then we say that M rejects w.

Definition 2.2.3 Let M = (Q, Σ, δ, q, F) be a finite automaton. The language L(M) accepted by M is defined to be the set of all strings that are accepted by M:

L(M) = {w : w is a string over Σ and M accepts w }.

Definition 2.2.4 A language A is called regular, if there exists a finite automaton M such that A = L(M).

We finish this section, by presenting an equivalent way of defining the language accepted by a finite automaton. Let M = (Q, Σ, δ, q, F) be a finite automaton. The transition function δ : Q × Σ → Q tells us that, when M is in state r ∈ Q and reads symbol a ∈ Σ, it switches from state r to state δ(r, a). Let Σ^{∗} denote the set of all strings over the alphabet Σ. (Σ^{∗} includes the empty string, which is denoted by ϵ.) We extend the function δ to a function

\overline{δ} : Q × Σ^{∗} → Q,

that is defined as follows. For any state r ∈ Q and for any string w over the alphabet Σ,

\overline{δ}(r, w) = \{ \begin{matrix}r & if w = ϵ, \\ δ(\overline{δ}(r, v), a) & if w = va, where v is a string and a ∈ Σ. \end{matrix}

What is the meaning of this function \overline{δ}? Let r be a state of Q, and let w be a string over the alphabet Σ. Then

• \overline{δ}(r, w) is the state that M reaches, when it starts in state r, reads the string w, and uses δ to switch from state to state.

Using this notation, we have

L(M) = \{w : w is a string over Σ and \overline{δ}(q, w) ∈ F \}.

2.2.1 An example of a finite automaton

Let

A = {w : w is a binary string containing an odd number of 1’s}.

We claim that this language A is regular. In order to prove this, we have to construct a finite automaton M such that A = L(M).

How to construct M? Here is a first idea: The finite automaton reads the input string w from left to right, and keeps track of the number of 1s it has seen. After having read the entire string w, it checks whether the number of 1s is odd (in which case w is accepted) or even (in which case w is rejected). Using this approach, the finite automaton needs a state for every integer i ≥ 0, indicating that the number of 1s read (up to that point) is equal to i. Hence, to design a finite automaton that follows this approach, we need an infinite number of states. But, the definition of finite automaton requires that the number of states is finite.

A better, and the correct approach, is to keep track of whether the number of 1s read is even or odd. This leads to the following finite automaton:

• The set of states is Q = {q_{e}, q_{o}}. If the finite automaton is in state q_{e}, then it has read an even number of 1s; if it is in state q_{o}, then it has read an odd number of 1s.

• The alphabet is Σ = {0, 1}.

• The start state is q_{e}, because at the start, the number of 1s read by the automaton is equal to 0, and 0 is even.

• The set F of accept states is F = {q_{o}}.

• The transition function δ is given by the following table:

\begin{aligned}& 0 & 1 \\ q_{e} & q_{e} & q_{o} \\ q_{o} & q_{o} & q_{e} \end{aligned}

This finite automaton M = (Q, Σ, δ, q_{e}, F) can also be described by its state diagram, which is given in the figure below. The arrow that comes “out of the blue” and enters the state q_{e}, indicates that q_{e} is the start state. The state depicted with double circles indicates the accept state.

q_{e}  q_{o} 0  0 11

We have constructed a finite automaton M that accepts the language A. Therefore, A is a regular language.

2.2.2 Another example of a finite automaton

Define the language A as

A = {w : w is a binary string containing 101 as a substring}.

Again, we claim that M is a regular language. In other words, we claim that there exists a finite automaton M that accepts A, i.e., A = L(M).

The finite automaton M will do the following, when reading an input string from left to right:

• It skips over all 0s, and stays in the start state.

• At the first 1, it switches to the state “maybe the next two symbols are 01”.

– If the next symbol is 1, then it stays in the state “maybe the next two symbols are 01”.

30 CHAPTER 2. FINITE AUTOMATA AND REGULAR LANGUAGES

– On the other hand, if the next symbol is 0, then it switches to the

state “maybe the next symbol is 1”.

∗ If the next symbol is indeed 1, then it switches to the accept state (but keeps on reading until the end of the string).

∗ On the other hand, if the next symbol is 0, then it switches

to the start state, and skips 0s until it reads 1 again.

By defining the following four states, this process will become clear:

• q_{1}: M is in this state if the last symbol read was 1, but the substring 101 has not been read.

• q_{10}: M is in this state if the last two symbols read were 10.

• q_{101}: M is in this state if the substring 101 has been read in the input string.

• q: In all other cases, M is in this state. Here is the formal description of the finite automaton that accepts the

language A:

• Q = {q, q_{1}, q_{10}, q_{101}},

• Σ = {0, 1},

• the start state is q,

• the set F of accept states is equal to F = {q_{101}}, and

• the transition function δ is given by the following table:

0  1
q  q q_{1}
q_{1} q_{10} q_{1}
q_{10} q q_{101}
q_{101} q_{101} q_{101}

The figure below gives the state diagram of the finite automaton M = (Q, Σ, δ, q_{e}, F).

q  q_{1} q_{10}  q_{101} 0  1  1 0  0  1  0, 1

2.3 Regular operations

In this section, we define three operations on languages. Later, we will answer the question whether the set of all regular languages is closed under these operations. Let X and Y be two languages over the same alphabet.

1. The union of X and Y is defined as

X ∪ Y = {w : w ∈ X or w ∈ Y }.

2. The concatenation of X and Y is defined as

XY = {ww′ : w ∈ X and w′ ∈ Y }.

In words, XY is the set of all strings obtained by taking an arbitrary string w in X and an arbitrary string w′ in Y, and gluing them together (such that w is to the left of w′).

3. The star of X is defined as

X^{∗} = {u_{1}u_{2} … u_{k} : k ≥ 0 and u_{i} ∈ X for all i = 1, 2, …, k}.

In words, X^{∗} is obtained by taking any finite number of strings in X, and gluing them together. Observe that k = 0 is allowed; this corresponds to the empty string ϵ. Thus, ϵ ∈ X^{∗}.

To give an example, let X = {A, C} and Y = {T, G}. Then

X ∪ Y = {A, C, T, G},

XY = {AT, AG, CT, CG},

and

X^{∗} = {ϵ, A, C, AA, CC, AC, CA, AAA, AAC, ACA, CAA, ACC, …}.

As another example, if Σ = {0, 1}, then Σ^{∗} is the set of all binary strings (including the empty string). Observe that a string always has a finite length.

Theorem 2.3.1 The set of regular languages is closed under the union operation, i.e., if X and Y are regular languages over the same alphabet, then X ∪ Y is also a regular language.

Proof. Since X and Y are regular languages, there are finite automata M_{1} = (Q_{1}, Σ, δ_{1}, q_{1}, F_{1}) and M_{2} = (Q_{2}, Σ, δ, q_{2}, F_{2}) that accept X and Y, respectively. In order to prove that X ∪ Y is regular, we have to construct a finite automaton M that accepts X ∪ Y. In other words, M must have the property that for every string w ∈ Σ^{∗},

M accepts w ⇔ M_{1} accepts w or M_{2} accepts w.

As a first idea, we may think that M could do the following:

• Starting in the start state q_{1}, M “runs” M_{1} on w.

• If, after having read w, M_{1} is in a state of F_{1}, then w ∈ X, thus w ∈ X ∪ Y and, therefore, M accepts w.

• On the other hand, if, after having read w, M_{1} is in a state that is not in F_{1}, then w ∉ X and M “runs” M_{2} on w, starting in the start state q_{2} of M_{2}. If, after having read w, M_{2} is in a state of F_{2}, then we know that w ∈ Y, thus w ∈ X ∪ Y and, therefore, M accepts w. Otherwise, we know that w ∉ X ∪ Y, and M rejects w.

This idea does not work, because the finite automaton M can read the input string w only once.

The correct approach is to run M_{1} and M_{2} simultaneously. We define the set Q of states of M to be the cross product Q_{1} × Q_{2}. If M is in state (r_{1}, r_{2}), then this means that

• if M_{1} would have read the input string up to this point, then it would be in state r_{1}, and • if M_{2} would have read the input string up to this point, then it would

be in state r_{2}.

This leads to the finite automaton M = (Q, Σ, δ, q, F), where

• Q = Q_{1} × Q_{2} = {(r_{1}, r_{2}) : r_{1} ∈ Q_{1} and r_{2} ∈ Q_{2}}. Observe that |Q| = |Q_{1}| × |Q_{2}|, which is finite.

• Σ is the alphabet of X and Y (recall that we assume that X and Y are languages over the same alphabet).

• The start state q of M is equal to q = (q_{1}, q_{2}).

• The set F of accept states of M is given by

F = {(r_{1}, r_{2}) : r_{1} ∈ F_{1} or r_{2} ∈ F_{2}} = (F_{1} × Q_{2}) ∪ (Q_{1} × F_{2}).

• The transition function δ : Q × Σ → Q is given by

δ((r_{1}, r_{2}), a) = (δ_{1}(r_{1}, a), δ_{2}(r_{2}, a)),

for all r_{1} ∈ Q_{1}, r_{2} ∈ Q_{2}, and a ∈ Σ.

To finish the proof, we have to show that this finite automaton M indeed accepts the language X ∪Y. Intuitively, this should be clear from the discussion above. The easiest way to give a formal proof is by using the extended transition functions \overline{δ_{1}} and \overline{δ_{2}}. (The extended transition function has been defined after Definition 2.2.4.) Here we go: Recall that we have to prove that

M accepts w ⇔ M_{1} accepts w or M_{2} accepts w,

i.e,

M accepts w ⇔ \overline{δ_{1}}(q_{1}, w) ∈ F_{1} or \overline{δ_{2}}(q_{2}, w) ∈ F_{2}.

In terms of the extended transition function \overline{δ} of the transition function δ of M, this becomes

\overline{δ}((q_{1}, q_{2}), w) ∈ F ⇔ \overline{δ_{1}}(q_{1}, w) ∈ F_{1} or \overline{δ_{2}}(q_{2}, w) ∈ F_{2}.  (2.1)

By applying the definition of the extended transition function, as given after Definition 2.2.4, to δ, it can be seen that

\overline{δ}((q_{1}, q_{2}), w) = (\overline{δ_{1}}(q_{1}, w), \overline{δ_{2}}(q_{2}, w)).

The latter equality implies that (2.1) is true and, therefore, M indeed accepts the language X ∪ Y.

What about the closure of the regular languages under the concatenation and star operations? It turns out that the regular languages are closed under these operations. But how do we prove this?

Let X and Y be two regular languages, and let M_{1} and M_{2} be finite automata that accept X and Y, respectively. How do we construct a finite automaton M that accepts the concatenation XY? Given an input string u, M has to decide whether or not u can be broken into two strings w and w′ (i.e., write u as u = ww′), such that w ∈ X and w′ ∈ Y. In words, M has to decide whether or not u can be broken into two substrings, such that the first substring is accepted by M_{1} and the second substring is accepted by M_{2}. The difficulty is caused by the fact that M has to make this decision by scanning the string u only once. If u ∈ XY, then M has to decide, during this single scan, where to break u into two substrings. Similarly, if u ∉ XY, then M has to decide, during this single scan, that u cannot be broken into two substrings such that the first substring is in X and the second substring is in Y.

It seems to be even more difficult to prove that X^{∗} is a regular language, if X itself is regular. In order to prove this, we need a finite automaton that, when given an arbitrary input string u, decides whether or not u can be broken into substrings such that each substring is in X. The problem is that, if u ∈ X^{∗}, the finite automaton has to determine into how many substrings, and where, the string u has to be broken; it has to do this during one single scan of the string u.

As we mentioned already, if X and Y are regular languages, then both XY and X^{∗} are also regular. In order to prove these claims, we will introduce a more general type of finite automaton.

The finite automata that we have seen until now are deterministic. This means the following:

• If the finite automaton M is in state r and if it reads the symbol a, then M switches from state r to the uniquely defined state δ(r, a).

From now on, we will call such a finite automaton a deterministic finite automaton (DFA). In the next section, we will define the notion of a nondeterministic finite automaton (NFA). For such an automaton, there are zero or more possible states to switch to. At first sight, nondeterministic finite automata seem to be more powerful than their deterministic counterparts. We will prove, however, that DFAs have the same power as NFAs. As we will see, using this fact, it will be easy to prove that the class of regular languages is closed under the concatenation and star operations.

2.4 Nondeterministic finite automata

We start by giving three examples of nondeterministic finite automata. These examples will show the difference between this type of automata and the deterministic versions that we have considered in the previous sections. After these examples, we will give a formal definition of a nondeterministic finite automaton.

2.4.1 A first example

Consider the following state diagram:

q_{1}  q_{2}  q_{3}  q_{4} 0, 1  1  0, ϵ  1  0, 1

You will notice three differences with the finite automata that we have seen until now. First, if the automaton is in state q_{1} and reads the symbol 1, then it has two options: Either it stays in state q_{1}, or it switches to state q_{2}. Second, if the automaton is in state q_{2}, then it can switch to state q_{3} without reading a symbol; this is indicated by the edge having the empty string ϵ as label. Third, if the automaton is in state q_{3} and reads the symbol 0, then it cannot continue.

Let us see what this automaton can do when it gets the string 010110 as input. Initially, the automaton is in the start state q_{1}.

• Since the first symbol in the input string is 0, the automaton stays in state q_{1} after having read this symbol.

• The second symbol is 1, and the automaton can either stay in state q_{1} or switch to state q_{2}.

– If the automaton stays in state q_{1}, then it is still in this state after having read the third symbol.

– If the automaton switches to state q_{2}, then it again has two options:

∗ Either read the third symbol in the input string, which is 0, and switch to state q_{3},

∗ or switch to state q_{3}, but do not read the third symbol.

If we continue in this way, then we see that, for the input string 010110, there are seven possible computations. All these computations are given in the figure below.

q_{1}  q_{1} 0  1 q_{1}  q_{1} 0  1 1  q_{1} q_{2} 1 1  q_{1} q_{2}  q_{1} 0 0 ϵ  q_{3} q_{3}  hang hang ϵ  q_{3}  q_{4} 1  0 q_{4} 1  q_{2} 0 ϵ  q_{3} q_{3}  hang 1  q_{4}  1  q_{4}  q_{4} 0

Consider the lowest path in the figure above:

• When reading the first symbol, the automaton stays in state q_{1}.

• When reading the second symbol, the automaton switches to state q_{2}.

• The automaton does not read the third symbol (equivalently, it “reads” the empty string ϵ), and switches to state q_{3}. At this moment, the automaton cannot continue: The third symbol is 0, but there is no edge leaving q_{3} that is labeled 0, and there is no edge leaving q_{3} that is labeled ϵ. Therefore, the computation hangs at this point.

From the figure, you can see that, out of the seven possible computations, exactly two end in the accept state q_{4} (after the entire input string 010110 has been read). We say that the automaton accepts the string 010110.

Now consider the input string 010. In this case, there are three possible computations:

1. q_{1} →^{0} q_{1} →^{1} q_{1} →^{0} q_{1}

2. q_{1} →^{0} q_{1} →^{1} q_{2} →^{0} q_{3}

3. q_{1} →^{0} q_{1} →^{1} q_{2} →^{ϵ} q_{3} → hang

None of these computations end in the accept state (after the entire input string 010 has been read). Therefore, we say that the automaton rejects the input string 010.

The state diagram given above is an example of a nondeterministic finite automaton (NFA). Informally, an NFA accepts a string, if there exists at least one path in the state diagram that (i) starts in the start state, (ii) does not hang before the entire string has been read, and (iii) ends in an accept state. A string for which (i), (ii), and (iii) does not hold is rejected by the NFA.

The NFA given above accepts all strings that contain 101 or 11 as a substring. All other strings are rejected.

2.4.2 A second example

Let A be the language

A = {w ∈ {0, 1}^{∗} : w has a 1 in the third position from the right}.

The following state diagram defines an NFA that accepts all strings in A, and rejects all strings that are not in A.

q_{1}  q_{2}  q_{3}  q_{4} 0, 1  1  0, 1  0, 1

This NFA does the following. If it is in the start state q_{1} and reads the symbol 1, then it either stays in state q_{1} or it “guesses” that this symbol is the third symbol from the right in the input string. In the latter case, the NFA switches to state q_{2}, and then it “verifies” that there are indeed exactly two remaining symbols in the input string. If there are more than two remaining symbols, then the NFA hangs (in state q_{4}) after having read the next two symbols.

Observe how this guessing mechanism is used: The automaton can only read the input string once, from left to right. Hence, it does not know when it reaches the third symbol from the right. When the NFA reads a 1, it can guess that this is the third symbol from the right; after having made this guess, it verifies whether or not the guess was correct.

At first sight, it seems difficult (or even impossible?) to construct a deterministic finite automaton (DFA) that accepts the language A: How does the DFA “know” that it has reached the third symbol from the right? It is, however, possible to construct such a DFA. This DFA uses eight states q_{ijk}, where i, j, and k range over all elements of {0, 1}. If the DFA is in state q_{ijk}, then

• the three most recently read symbols were ijk, or

• it has read only two symbols, which were jk, or

• it has read only one symbol, which was k.

The start state is q_{000}, and the set of accept states is equal to {q_{100}, q_{110}, q_{101}, q_{111}}. The transition function of the DFA is given by the following state diagram.

q_{000}  q_{100}  q_{010}  q_{110} q_{001}  q_{101}  q_{011}  q_{111} 0 1  0 1  0 1  0 1  0 1 10 1 0 1 0

2.4.3 A third example

Consider the following state diagram, which depicts an NFA whose alphabet is {0}.

ϵϵ  0 00  0 0

This NFA accepts the language

A = {0^{k} : k ≡ 0 mod 2 or k ≡ 0 mod 3}.

Observe that A is the union of the two languages

A_{1} = {0^{k} : k ≡ 0 mod 2}

and

A_{2} = {0^{k} : k ≡ 0 mod 3}.

The NFA basically consists of two DFAs: one of these accepts A_{1}, whereas the other accepts A_{2}. Given an input string w, the NFA has to decide whether or not w ∈ A, which is equivalent to deciding whether or not w ∈ A_{1} or w ∈ A_{2}. The NFA makes this decision in the following way: At the start, it “guesses” whether to check whether or not w ∈ A_{1} (i.e., the length of w is even), or to check whether or not w ∈ A_{2} (i.e., the length of w is a multiple of 3). After having made the guess, it verifies whether or not the guess was correct. If w ∈ A, then there is a way of making the correct guess and verifying that w is indeed an element of A (by ending in an accept state). If w ∉ A, then no matter which guess is made, the NFA will never end in an accept state.

2.4.4 Definition of nondeterministic finite automaton

The previous examples give you an idea what nondeterministic finite automata are, and how they work. In this section, we give a formal definition of these automata.

For any alphabet Σ, we define Σ_{ϵ} to be the set

Σ_{ϵ} = Σ ∪ {ϵ}.

Recall the notion of a power set: For any set Q, the power set of Q, denoted by P(Q), is the set of all subsets of Q, i.e.,

P(Q) = {S : S ⊆ Q}.

Definition 2.4.1 A nondeterministic finite automaton (NFA) is a 5-tuple M = (Q, Σ, δ, q, F) where

1. Q is a finite set, whose elements are called states,

2. Σ is a finite set, called the alphabet; the elements of Σ are called symbols,

3. δ : Q × Σ_{ϵ} → P(Q) is a function, called the transition function,

4. q is an element of Q; it is called the start state,

5. F is a subset of Q; the elements of F are called accept states.

As for DFAs, the transition function δ can be thought of as the “program” of the finite automaton M = (Q, Σ, δ, q, F):

• Let r ∈ Q, and let a ∈ Σ_{ϵ}. Then δ(r, a) is a (possibly empty) subset of Q. If the NFA M is in state r, and if it reads a (where a may be the empty string ϵ), then M can switch from state r to any state in δ(r, a). If δ(r, a) = ∅, then M cannot continue and the computation hangs.

The example given in Section 2.4.1 is an NFA, where Q = {q_{1}, q_{2}, q_{3}, q_{4}}, Σ = {0, 1}, the start state is q_{1}, the set of accept states is F = {q_{4}}, and the transition function δ is given by the following table:

0  1  ϵ
q_{1} {q_{1}} {q_{1}, q_{2}} ∅
q_{2} {q_{3}}  ∅  {q_{3}}
q_{3} ∅  {q_{4}}  ∅
q_{4} {q_{4}} {q_{4}}  ∅

Definition 2.4.2 Let M = (Q, Σ, δ, q, F) be an NFA, and let w ∈ Σ^{∗}. We say that M accepts w, if w can be written as w = y_{1}y_{2} … y_{m}, where y_{i} ∈ Σ_{ϵ} for all i with 1 ≤ i ≤ m, and if there exists a sequence r_{0}, r_{1}, …, r_{m} of states in Q, such that

• r_{0} = q,

• r_{i+1} ∈ δ(r_{i}, y_{i+1}), for i = 0, 1, …, m − 1, and

• r_{m} ∈ F.

Otherwise, we say that M rejects the string w.

The NFA in the example in Section 2.4.1 accepts the string 01100. This can be seen by taking

• w = 01ϵ100, and

• r_{0} = q_{1}, r_{1} = q_{1}, r_{2} = q_{2}, r_{3} = q_{3}, r_{4} = q_{4}, r_{5} = q_{4}, and r_{6} = q_{4}.

Definition 2.4.3 Let M = (Q, Σ, δ, q, F) be an NFA. The language L(M) accepted by M is defined as

L(M) = {w ∈ Σ^{∗} : M accepts w }.

2.5 Equivalence of DFAs and NFAs

You may have the impression that nondeterministic finite automata are more powerful than deterministic finite automata. In this section, we will show that this is not the case. That is, we will prove that a language can be accepted by a DFA if and only if it can be accepted by an NFA. In order to prove this, we will show how to convert an arbitrary NFA to a DFA that accepts the same language.

What about converting a DFA to an NFA? Well, there is (almost) nothing to do, because a DFA is also an NFA. This is not quite true, because

• the transition function of a DFA maps a state and a symbol to a state, whereas

• the transition function of an NFA maps a state and a symbol to a set of zero or more states.

The formal conversion of a DFA to an NFA is done as follows: Let M = (Q, Σ, δ, q, F) be a DFA. Recall that δ is a function δ : Q × Σ → Q. We define the function δ′ : Q × Σ_{ϵ} → P(Q) as follows. For any r ∈ Q and for any a ∈ Σ_{ϵ},

δ′(r, a) = \{ \begin{matrix}\{δ(r, a)\} & if a ≠ ϵ, \\ ∅ & if a = ϵ. \end{matrix}

Then N = (Q, Σ, δ′, q, F) is an NFA, whose behavior is exactly the same as that of the DFA M; the easiest way to see this is by observing that the state diagrams of M and N are equal. Therefore, we have L(M) = L(N).

In the rest of this section, we will show how to convert an NFA to a DFA:

Theorem 2.5.1 Let N = (Q, Σ, δ, q, F) be a nondeterministic finite automaton. There exists a deterministic finite automaton M, such that L(M) = L(N).

Proof. Recall that the NFA N can (in general) perform more than one computation on a given input string. The idea of the proof is to construct a DFA M that runs all these different computations simultaneously. (We have seen this idea already in the proof of Theorem 2.3.1.) To be more precise, the DFA M will have the following property:

• the state that M is in after having read an initial part of the input string corresponds exactly to the set of all states that N can reach after having read the same part of the input string.

We start by presenting the conversion for the case when N does not contain ϵ-transitions. In other words, the state diagram of N does not contain any edge that has ϵ as a label. (Later, we will extend the conversion to the general case.) Let the DFA M be defined as M = (Q′, Σ, δ′, q′, F′), where

• the set Q′ of states is equal to Q′ = P(Q); observe that |Q′| = 2^{|Q|},

• the start state q′ is equal to q′ = {q}; so M has the “same” start state as N,

• the set F′ of accept states is equal to the set of all elements R of Q′ having the property that R contains at least one accept state of N, i.e.,

F′ = {R ∈ Q′ : R ∩ F ≠ ∅},

• the transition function δ′ : Q′ × Σ → Q′ is defined as follows: For each R ∈ Q′ and for each a ∈ Σ,

δ′(R, a) = ⋃_{r∈R} δ(r, a).

Let us see what the transition function δ′ of M does. First observe that, since N is an NFA, δ(r, a) is a subset of Q. This implies that δ′(R, a) is the union of subsets of Q and, therefore, is also a subset of Q. Hence, δ′(R, a) is an element of Q′.

The set δ(r, a) is equal to the set of all states of the NFA N that can be reached from state r, by reading the symbol a. For each r ∈ R, we take the union of these sets δ(r, a) to obtain the new set δ′(R, a). This new set is the state that the DFA M reaches from state R, by reading the symbol a.

In this way, we obtain the correspondence that was given in the beginning of this proof.

After this warming-up, we can consider the general case. In other words, we allow that there are ϵ-transitions in the NFA N. The DFA M is defined as above, except that the transition function δ′ has to be modified.

We need the notion of ϵ-closure. For any state r of the NFA N, the ϵclosure of r, denoted by C_{ϵ}(r), is defined to be the set of all states in N that can be reached from r, by making zero or more ϵ-transitions. For any state R of the DFA M (hence, R ⊆ Q), we define

C_{ϵ}(R) = ⋃_{r∈R} C_{ϵ}(r).

How do we define the transition function δ′ of the DFA M? Assume that M is in state R, and reads the symbol a. At this moment, the NFA N would have been in any state r of R. By reading the symbol a, N can switch to any state in δ(r, a), and then make zero or more ϵ-transitions. Hence, the NFA can switch to any state in the set C_{ϵ}(δ(r, a)). Based on this, we define δ′(R, a) to be

δ′(R, a) = ⋃_{r∈R} C_{ϵ}(δ(r, a)).

To summarize, the NFA N = (Q, Σ, δ, q, F) is converted to the DFA M = (Q′, Σ, δ′, q′, F′), where

• Q′ = P(Q), • q′ = C_{ϵ}({q}),

• F′ = {R ∈ Q′ : R ∩ F ≠ ∅},

• δ′ : Q′ × Σ → Q′ is defined as follows: For each R ∈ Q′ and for each

a ∈ Σ,

δ′(R, a) = ⋃_{r∈R} C_{ϵ}(δ(r, a)).

The results proved until now can be summarized in the following theorem.

Theorem 2.5.2 Let A be a language. Then A is regular if and only if there exists a nondeterministic finite automaton that accepts A.

2.5.1 An example

Consider the NFA N = (Q, Σ, δ, q, F), where Q = {1, 2, 3}, Σ = {a, b}, q = 1, F = {2}, and δ is given by the following table:

a  b  ϵ
1 {3} ∅ {2}
2 {1} ∅  ∅
3 {2} {2, 3} ∅

The state diagram of N is as follows:

1  2 3 a  a ϵb  a, b

We will show how to convert this NFA N to a DFA M that accepts the same language. Following the proof of Theorem 2.5.1, the DFA M is specified by M = (Q′, Σ, δ′, q′, F′), where each of the components is defined below.

• Q′ = P(Q). Hence,

Q′ = {∅, {1}, {2}, {3}, {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}}.

• q′ = C_{ϵ}({q}). Hence, the start state q′ of M is the set of all states of N that can be reached from N’s start state q = 1, by making zero or more ϵ-transitions. We obtain

q′ = C_{ϵ}({q}) = C_{ϵ}({1}) = {1, 2}.

• F′ = {R ∈ Q′ : R ∩ F ≠ ∅}. Hence, the accept states of M are those states that contain the accept state 2 of N. We obtain

F′ = {{2}, {1, 2}, {2, 3}, {1, 2, 3}}.

• δ′ : Q′ × Σ → Q′ is defined as follows: For each R ∈ Q′ and for each

a ∈ Σ,

δ′(R, a) = ⋃_{r∈R} C_{ϵ}(δ(r, a)).

In this example δ′ is given by

δ′(∅, a) = ∅  δ′(∅, b) = ∅
δ′({1}, a) = {3}  δ′({1}, b) = ∅
δ′({2}, a) = {1, 2}  δ′({2}, b) = ∅
δ′({3}, a) = {2} δ′({3}, b) = {2, 3}
δ′({1, 2}, a) = {1, 2, 3}  δ′({1, 2}, b) = ∅
δ′({1, 3}, a) = {2, 3} δ′({1, 3}, b) = {2, 3}
δ′({2, 3}, a) = {1, 2} δ′({2, 3}, b) = {2, 3}
δ′({1, 2, 3}, a) = {1, 2, 3} δ′({1, 2, 3}, b) = {2, 3}

The state diagram of the DFA M is as follows:

∅  {1} {2}  {3} {1, 2}  {2, 3}  {1, 3} {1, 2, 3} a, b  b  a b a  a b a, b ab  b a ba

We make the following observations:

• The states {1} and {1, 3} do not have incoming edges. Therefore, these two states cannot be reached from the start state {1, 2}.

• The state {3} has only one incoming edge; it comes from the state {1}. Since {1} cannot be reached from the start state, {3} cannot be reached from the start state.

• The state {2} has only one incoming edge; it comes from the state {3}. Since {3} cannot be reached from the start state, {2} cannot be reached from the start state.

Hence, we can remove the four states {1}, {2}, {3}, and {1, 3}. The resulting DFA accepts the same language as the DFA above. This leads to the following state diagram, which depicts a DFA that accepts the same language as the NFA N:

∅ {1, 2}  {2, 3} {1, 2, 3} a, b ab  b a ba

2.6 Closure under the regular operations

In Section 2.3, we have defined the regular operations union, concatenation, and star. We proved in Theorem 2.3.1 that the union of two regular languages is a regular language. We also explained why it is not clear that the concatenation of two regular languages is regular, and that the star of a regular language is regular. In this section, we will see that the concept of NFA, together with Theorem 2.5.2, can be used to give a simple proof of the fact that the regular languages are indeed closed under the regular operations. We start by giving an alternative proof of Theorem 2.3.1:

Theorem 2.6.1 The set of regular languages is closed under the union operation, i.e., if A_{1} and A_{2} are regular languages over the same alphabet, then A_{1} ∪ A_{2} is also a regular language.

q_{1}  M_{1} M_{2} q_{2}  q_{0}  q_{1} q_{2} ϵϵ  M

Figure 2.1: The NFA that accepts L(M_{1}) ∪ L(M_{2}).

Proof. Since A_{1} is regular, there is, by Theorem 2.5.2, an NFA M_{1} = (Q_{1}, Σ, δ_{1}, q_{1}, F_{1}), such that A_{1} = L(M_{1}). Similarly, there is an NFA M_{2} = (Q_{2}, Σ, δ_{2}, q_{2}, F_{2}), such that A_{2} = L(M_{2}). We may assume that Q_{1} ∩ Q_{2} = ∅. From these two NFAs, we will construct an NFA M = (Q, Σ, δ, q_{0}, F), such that L(M) = A_{1} ∪ A_{2}. The construction is illustrated in Figure 2.1. The NFA M is defined as follows:

1. Q = {q_{0}} ∪ Q_{1} ∪ Q_{2}, where q_{0} is a new state.

2. q_{0} is the start state of M.

3. F = F_{1} ∪ F_{2}.

4. δ : Q × Σ_{ϵ} → P(Q) is defined as follows: For any r ∈ Q and for any a ∈ Σ_{ϵ},

δ(r, a) = \{ \begin{matrix}δ_{1}(r, a) & if r ∈ Q_{1}, \\ δ_{2}(r, a) & if r ∈ Q_{2}, \\ \{q_{1}, q_{2}\} & if r = q_{0} and a = ϵ, \\ ∅ & if r = q_{0} and a ≠ ϵ. \end{matrix}

Theorem 2.6.2 The set of regular languages is closed under the concatenation operation, i.e., if A_{1} and A_{2} are regular languages over the same alphabet, then A_{1}A_{2} is also a regular language.

Proof. Let M_{1} = (Q_{1}, Σ, δ_{1}, q_{1}, F_{1}) be an NFA, such that A_{1} = L(M_{1}). Similarly, let M_{2} = (Q_{2}, Σ, δ_{2}, q_{2}, F_{2}) be an NFA, such that A_{2} = L(M_{2}). We may assume that Q_{1}∩Q_{2} = ∅. We will construct an NFA M = (Q, Σ, δ, q_{0}, F), such that L(M) = A_{1}A_{2}. The construction is illustrated in Figure 2.2. The NFA M is defined as follows:

1. Q = Q_{1} ∪ Q_{2}.

2. q_{0} = q_{1}.

3. F = F_{2}.

4. δ : Q × Σ_{ϵ} → P(Q) is defined as follows: For any r ∈ Q and for any a ∈ Σ_{ϵ},

δ(r, a) = \{ \begin{matrix}δ_{1}(r, a) & if r ∈ Q_{1} and r ∉ F_{1}, \\ δ_{1}(r, a) & if r ∈ F_{1} and a ≠ ϵ, \\ δ_{1}(r, a) ∪ \{q_{2}\} & if r ∈ F_{1} and a = ϵ, \\ δ_{2}(r, a) & if r ∈ Q_{2}. \end{matrix}

q_{1}  M_{1}  M_{2} q_{2} q_{2} ϵ ϵϵ q_{0}  M

Figure 2.2: The NFA that accepts L(M_{1})L(M_{2}).

Theorem 2.6.3 The set of regular languages is closed under the star operation, i.e., if A is a regular language, then A^{∗} is also a regular language.

Proof. Let N = (Q_{1}, Σ, δ_{1}, q_{1}, F_{1}) be an NFA, such that A = L(N). We will construct an NFA M = (Q, Σ, δ, q_{0}, F), such that L(M) = A^{∗}. The construction is illustrated in Figure 2.3. The NFA M is defined as follows:

1. Q = {q_{0}} ∪ Q_{1}, where q_{0} is a new state.

2. q_{0} is the start state of M.

q_{1}  N  q_{1} q_{0} ϵ  ϵ ϵ ϵ  M

Figure 2.3: The NFA that accepts (L(N))^{∗}.

3. F = {q_{0}} ∪ F_{1}. (Since ϵ ∈ A^{∗}, q_{0} has to be an accept state.)

4. δ : Q × Σ_{ϵ} → P(Q) is defined as follows: For any r ∈ Q and for any a ∈ Σ_{ϵ},

δ(r, a) = \{ \begin{matrix}δ_{1}(r, a) & if r ∈ Q_{1} and r ∉ F_{1}, \\ δ_{1}(r, a) & if r ∈ F_{1} and a ≠ ϵ, \\ δ_{1}(r, a) ∪ \{q_{1}\} & if r ∈ F_{1} and a = ϵ, \\ \{q_{1}\} & if r = q_{0} and a = ϵ, \\ ∅ & if r = q_{0} and a ≠ ϵ. \end{matrix}

In the final theorem of this section, we mention (without proof) two more closure properties of the regular languages:

Theorem 2.6.4 The set of regular languages is closed under the intersection and the complement operations:

1. If A is a regular language over the alphabet Σ, then the complement

\overline{A} = \{w ∈ Σ^{∗} : w ∉ A\}

is also a regular language.

2. If A_{1} and A_{2} are regular languages over the same alphabet, then the intersection

A_{1} ∩ A_{2} = {w ∈ Σ^{∗} : w ∈ A_{1} and w ∈ A_{2}}

is also a regular language.

2.7 Regular expressions

In this section, we present regular expressions, which are a means to describe languages. As we will see, the class of languages that can be described by regular expressions coincides with the class of regular languages.

Before formally defining the notion of a regular expression, we give some examples. Consider the expression

(0 ∪ 1)01^{∗}.

The language described by this expression is the set of all binary strings

1. that start with either 0 or 1 (this is indicated by (0 ∪ 1)),

2. for which the second symbol is 0 (this is indicated by 0), and

3. that end with zero or more 1s (this is indicated by 1^{∗}).

That is, the language described by this expression is

{00, 001, 0011, 00111, …, 10, 101, 1011, 10111, …}.

Here are some more examples (where the alphabet is {0, 1}):

The language {w : w contains exactly two 0s} is described by the expression

1^{∗}01^{∗}01^{∗}.

The language {w : w contains at least two 0s} is described by the expression

(0 ∪ 1)^{∗}0(0 ∪ 1)^{∗}0(0 ∪ 1)^{∗}.

The language {w : 1011 is a substring of w} is described by the expression

(0 ∪ 1)^{∗}1011(0 ∪ 1)^{∗}.

The language {w : the length of w is even} is described by the expression

((0 ∪ 1)(0 ∪ 1))^{∗} .

The language {w : the length of w is odd} is described by the expression

(0 ∪ 1) ((0 ∪ 1)(0 ∪ 1))^{∗} .

The language {1011, 0} is described by the expression

1011 ∪ 0.

The language {w : the first and last symbols of w are equal} is described by the expression

0(0 ∪ 1)^{∗}0 ∪ 1(0 ∪ 1)^{∗}1 ∪ 0 ∪ 1.

After these examples, we give a formal (and inductive) definition of regular expressions:

Definition 2.7.1 Let Σ be a non-empty alphabet.

1. ϵ is a regular expression.

2. ∅ is a regular expression.

3. For each a ∈ Σ, a is a regular expression.

4. If R_{1} and R_{2} are regular expressions, then R_{1} ∪ R_{2} is a regular expression.

5. If R_{1} and R_{2} are regular expressions, then R_{1}R_{2} is a regular expression.

6. If R is a regular expression, then R^{∗} is a regular expression.

You can regard 1., 2., and 3. as being the “building blocks” of regular expressions. Items 4., 5., and 6. give rules that can be used to combine regular expressions into a new (and “larger”) regular expression. To give an example, we claim that

(0 ∪ 1)^{∗}101(0 ∪ 1)^{∗}

is a regular expression. In order to prove this, we have to show that this expression can be “built” using the “rules” given in Definition 2.7.1. Here we go:

54 CHAPTER 2. FINITE AUTOMATA AND REGULAR LANGUAGES • By 3., 0 is a regular expression.

• By 3., 1 is a regular expression.

• Since 0 and 1 are regular expressions, by 4., 0∪1 is a regular expression.

• Since 0∪1 is a regular expression, by 6., (0∪1)^{∗} is a regular expression.

• Since 1 and 0 are regular expressions, by 5., 10 is a regular expression.

• Since 10 and 1 are regular expressions, by 5., 101 is a regular expression.

• Since (0 ∪ 1)^{∗} and 101 are regular expressions, by 5., (0 ∪ 1)^{∗}101 is a regular expression.

• Since (0 ∪ 1)^{∗}101 and (0 ∪ 1)^{∗} are regular expressions, by 5., (0 ∪ 1)^{∗}101(0 ∪ 1)^{∗} is a regular expression.

Next we define the language that is described by a regular expression: Definition 2.7.2 Let Σ be a non-empty alphabet.

1. The regular expression ϵ describes the language {ϵ}.

2. The regular expression ∅ describes the language ∅.

3. For each a ∈ Σ, the regular expression a describes the language {a}.

4. Let R_{1} and R_{2} be regular expressions, and let L_{1} and L_{2} be the languages described by them, respectively. The regular expression R_{1} ∪R_{2} describes the language L_{1} ∪ L_{2}.

5. Let R_{1} and R_{2} be regular expressions, and let L_{1} and L_{2} be the languages described by them, respectively. The regular expression R_{1}R_{2} describes the language L_{1}L_{2}.

6. Let R be a regular expression, and let L be the language described by

it. The regular expression R^{∗} describes the language L^{∗}. We consider some examples:

• The regular expression (0∪ϵ)(1∪ϵ) describes the language {01, 0, 1, ϵ}.

2.8. EQUIVALENCE OF REGULAR EXPRESSIONS AND REGULAR LANGUAGES55

• The regular expression 0 ∪ ϵ describes the language {0, ϵ}, whereas the regular expression 1^{∗} describes the language {ϵ, 1, 11, 111, …}. Therefore,the regular expression (0 ∪ ϵ)1^{∗} describes the language

{0, 01, 011, 0111, …, ϵ, 1, 11, 111, …}.

Observe that this language is also described by the regular expression 01^{∗} ∪ 1^{∗}.

• The regular expression 1^{∗}∅ describes the language ∅.

• The regular expression ∅^{∗} describes the language {ϵ}.

Definition 2.7.3 Let R_{1} and R_{2} be regular expressions, and let L_{1} and L_{2} be the languages described by them, respectively. If L_{1} = L_{2} (i.e., R_{1} and R_{2} describe the same language), then we will write R_{1} = R_{2}.

Hence, even though (0∪ϵ)1^{∗} and 01^{∗}∪1^{∗} are different regular expressions, we write

(0 ∪ ϵ)1^{∗} = 01^{∗} ∪ 1^{∗},

because they describe the same language.

2.8 Equivalence of regular expressions and regular languages

In the beginning of Section 2.7, we mentioned the following result:

Theorem 2.8.1 Let L be a language. Then L is regular if and only if there exists a regular expression that describes L.

The proof of this theorem consists of two parts:

• In Section 2.8.1, we will prove that every regular expression describes a regular language.

• In Section 2.8.2, we will prove that every DFA M can be converted to a regular expression that describes the language L(M).

These two results will prove Theorem 2.8.1.

2.8.1 Every regular expression describes a regular language

Let R be an arbitrary regular expression over the alphabet Σ. We will prove that the language described by R is a regular language. The proof is by induction on the structure of R (i.e., by induction on the way R is “built” using the “rules” given in Definition 2.7.1).

The first base case: Assume that R = ϵ. Then R describes the language {ϵ}. In order to prove that this language is regular, it suffices, by Theorem 2.5.2, to construct an NFA M = (Q, Σ, δ, q, F) that accepts this language. This NFA is obtained by defining Q = {q}, q is the start state, F = {q}, and δ(q, a) = ∅ for all a ∈ Σ_{ϵ}. The figure below gives the state diagram of M:

q

The second base case: Assume that R = ∅. Then R describes the language ∅. In order to prove that this language is regular, it suffices, by Theorem 2.5.2, to construct an NFA M = (Q, Σ, δ, q, F) that accepts this language. This NFA is obtained by defining Q = {q}, q is the start state, F = ∅, and δ(q, a) = ∅ for all a ∈ Σ_{ϵ}. The figure below gives the state diagram of M:

q

The third base case: Let a ∈ Σ, and assume that R = a. Then R describes the language {a}. In order to prove that this language is regular, it suffices, by Theorem 2.5.2, to construct an NFA M = (Q, Σ, δ, q_{1}, F) that accepts this language. This NFA is obtained by defining Q = {q_{1}, q_{2}}, q_{1} is the start state, F = {q_{2}}, and

δ(q_{1}, a) = {q_{2}}
δ(q_{1}, b) = ∅ for all b ∈ Σ_{ϵ} \ {a}
δ(q_{2}, b) = ∅ for all b ∈ Σ_{ϵ}

The figure below gives the state diagram of M:

q_{1}  a  q_{2}

The first case of the induction step: Assume that R = R_{1} ∪ R_{2}, where R_{1} and R_{2} are regular expressions. Let L_{1} and L_{2} be the languages described by R_{1} and R_{2}, respectively, and assume that L_{1} and L_{2} are regular. Then R describes the language L_{1} ∪ L_{2}, which, by Theorem 2.6.1, is regular.

The second case of the induction step: Assume that R = R_{1}R_{2}, where R_{1} and R_{2} are regular expressions. Let L_{1} and L_{2} be the languages described by R_{1} and R_{2}, respectively, and assume that L_{1} and L_{2} are regular. Then R describes the language L_{1}L_{2}, which, by Theorem 2.6.2, is regular.

The third case of the induction step: Assume that R = (R_{1})^{∗}, where R_{1} is a regular expression. Let L_{1} be the language described by R_{1}, and assume that L_{1} is regular. Then R describes the language (L_{1})^{∗}, which, by Theorem 2.6.3, is regular.

This concludes the proof of the claim that every regular expression describes a regular language.

To give an example, consider the regular expression

(ab ∪ a)^{∗},

where the alphabet is {a, b}. We will prove that this regular expression describes a regular language, by constructing an NFA that accepts the language described by this regular expression. Observe how the regular expression is “built”:

• Take the regular expressions a and b, and combine them into the regular expression ab.

• Take the regular expressions ab and a, and combine them into the regular expression ab ∪ a.

• Take the regular expressions ab ∪ a, and transform it into the regular expression (ab ∪ a)^{∗}.

First, we construct an NFA M_{1} that accepts the language described by the regular expression a:

\begin{aligned}& a \\ M_{1} \end{aligned}

Next, we construct an NFA M_{2} that accepts the language described by the regular expression b:

M_{2}  b

Next, we apply the construction given in the proof of Theorem 2.6.2 to M_{1} and M_{2}. This gives an NFA M_{3} that accepts the language described by the regular expression ab:

M_{3}  a  ϵ  b

Next, we apply the construction given in the proof of Theorem 2.6.1 to M_{3} and M_{1}. This gives an NFA M_{4} that accepts the language described by the regular expression ab ∪ a:

a  ϵ  b a ϵϵ M_{4}

Finally, we apply the construction given in the proof of Theorem 2.6.3 to M_{4}. This gives an NFA M_{5} that accepts the language described by the regular expression (ab ∪ a)^{∗}:

a  ϵ  b a ϵϵ ϵ  ϵ ϵ M_{5}

2.8.2 Converting a DFA to a regular expression

In this section, we will prove that every DFA M can be converted to a regular expression that describes the language L(M). In order to prove this result, we need to solve recurrence relations involving languages.

Solving recurrence relations

Let Σ be an alphabet, and let B, C, and L be languages in Σ^{∗} such that ϵ ∉ B, and

L = BL ∪ C.

Can we “solve” this equation for L? That is, can we express L in terms of B and C?

Let u be an arbitrary string in L, and let us determine how u looks like. Since u ∈ L and L = BL ∪ C, we know that u is a string in BL ∪ C. Hence, there are two possibilities for u.

1. u is an element of C.

2. u is an element of BL. In this case, there are strings b ∈ B and v ∈ L such that u = bv. Since v is a string in L, it is also a string in BL ∪ C. Hence, there are two possibilities for v.

(a) v is an element of C. In this case,

u = bv, where b ∈ B and v ∈ C.

(b) v is an element of BL. In this case, there are strings b′ ∈ B and w ∈ L such that v = b′w. Since w is a string in L, it is also a string in BL ∪ C. Hence, there are two possibilities for w.

i. w is an element of C. In this case,

u = bb′w, where b, b′ ∈ B and w ∈ C.

ii. w is an element of BL. In this case, there are strings b′′ ∈ B and x ∈ L such that w = b′′x. Since x is a string in L, it is also a string in BL ∪ C. Hence, there are two possibilities for x. A. x is an element of C. In this case,

u = bb′b′′x, where b, b′, b′′ ∈ B and x ∈ C.

B. x is an element of BL. Etc., etc.

This process hopefully convinces you that any string u in L can be written as the concatenation of zero or more strings in B, followed by one string in C. In fact, L consists of exactly those strings having this property:

Lemma 2.8.2 Let Σ be an alphabet, and let B, C, and L be languages in Σ^{∗} such that ϵ ∉ B and

L = BL ∪ C.

Then

L = B^{∗}C.

Proof. First, we show that B^{∗}C ⊆ L. Let u be an arbitrary string in B^{∗}C. Then u is the concatenation of k strings of B, for some k ≥ 0, followed by one string of C. We proceed by induction on k.

The base case is when k = 0. In this case, u is a string in C. (Why? Since u ∈ B^{∗}C, and we are choosing k = 0 strings from B.) Hence, u is a string in BL ∪ C. Since BL ∪ C = L, it follows that u is a string in L.

Now let k ≥ 1. Then we can write u = vwc, where v is a string in B, w is the concatenation of k − 1 strings of B, and c is a string of C. Define y = wc. Observe that y is the concatenation of k − 1 strings of B followed by one string of C. Therefore, by induction, the string y is an element of L. Hence, u = vy, where v is a string in B and y is a string in L. This shows that u is a string in BL. Hence, u is a string in BL ∪ C. Since BL ∪ C = L, it follows that u is a string in L. This completes the proof that B^{∗}C ⊆ L.

It remains to show that L ⊆ B^{∗}C. Let u be an arbitrary string in L, and let ℓ be its length (i.e., ℓ is the number of symbols in u). We prove by induction on ℓ that u is a string in B^{∗}C.

The base case is when ℓ = 0. Then u = ϵ. Since u ∈ L and L = BL ∪ C, u is a string in BL ∪ C. Since ϵ ∉ B, u cannot be a string in BL. Hence, u must be a string in C. Since C ⊆ B^{∗}C, it follows that u is a string in B^{∗}C.

Let ℓ ≥ 1. If u is a string in C, then u is a string in B^{∗}C and we are done. So assume that u is not a string in C. Since u ∈ L and L = BL ∪ C, u is a string in BL. Hence, there are strings b ∈ B and v ∈ L such that u = bv. Since ϵ ∉ B, the length of b is at least one; hence, the length of v is less than the length of u. By induction, v is a string in B^{∗}C. Hence, u = bv, where b ∈ B and v ∈ B^{∗}C. This shows that u ∈ B(B^{∗}C). Since B(B^{∗}C) ⊆ B^{∗}C, it follows that u ∈ B^{∗}C.

The conversion

We will now use Lemma 2.8.2 to prove that every DFA can be converted to a regular expression.

Let M = (Q, Σ, δ, q, F) be an arbitrary deterministic finite automaton. We will show that there exists a regular expression that describes the language L(M).

For each state r ∈ Q, we define

L_{r} = {w ∈ Σ^{∗} : the path in the state diagram of M that starts

in state r and that corresponds to w ends in a state of F }.

We will show that each such language L_{r} can be described by a regular expression. Since L(M) = L_{q}, this will prove that L(M) can be described by a regular expression.

The basic idea is to set up equations for the languages L_{r}, which we then solve using Lemma 2.8.2. We claim that

L_{r} = ⋃_{a∈Σ} a · L_{δ(r,a)}  if r ∉ F .  (2.2)

Why is this true? Let w be a string in L_{r}. Then the path P in the state diagram of M that starts in state r and that corresponds to w ends in a state of F. Since r ∉ F, this path contains at least one edge. Let r′ be the state that follows the first state (i.e., r) of P. Then r′ = δ(r, b) for some symbol b ∈ Σ. Hence, b is the first symbol of w. Write w = bv, where v is the remaining part of w. Then the path P′ = P \ {r} in the state diagram of M that starts in state r′ and that corresponds to v ends in a state of F. Therefore, v ∈ L_{r′} = L_{δ(r,b)}. Hence,

w ∈ b · L_{δ(r,b)} ⊆ ⋃_{a∈Σ} a · L_{δ(r,a)}.

Conversely, let w be a string in ⋃_{a∈Σ} a · L_{δ(r,a)}. Then there is a symbol b ∈ Σ and a string v ∈ L_{δ(r,b)} such that w = bv. Let P′ be the path in the state diagram of M that starts in state δ(r, b) and that corresponds to v. Since v ∈ L_{δ(r,b)}, this path ends in a state of F. Let P be the path in the state diagram of M that starts in r, follows the edge to δ(r, b) and then follows P′. This path P ends in a state of F. Therefore w ∈ L_{r}. This proves the correctness of (2.2).

Similarly, we can prove that

L_{r} = ϵ ∪ (⋃_{a∈Σ} a · L_{δ(r,a)})  if r ∈ F .  (2.3)

So we now have a set of equations in the “unknowns” L_{r}, for r ∈ Q. The number of equations is equal to the size of Q. The regular expression for L(M) = L_{q} is obtained by solving these equations using Lemma 2.8.2.

An example

Consider the deterministic finite automaton M = (Q, Σ, δ, q_{0}, F), where Q = {q_{0}, q_{1}, q_{2}}, Σ = {a, b}, q_{0} is the start state, F = {q_{2}}, and δ is given in the state diagram below. We show how to obtain the regular expression that describes the language accepted by M.

q_{0}  q_{1}  q_{2} a  a a b b b

For this case, (2.2) and (2.3) give the following equations.

\{ \begin{matrix}L_{q_{0}} & = & a · L_{q_{0}} ∪ b · L_{q_{2}} \\ L_{q_{1}} & = & a · L_{q_{0}} ∪ b · L_{q_{1}} \\ L_{q_{2}} & = & ϵ ∪ a · L_{q_{1}} ∪ b · L_{q_{0}} \end{matrix}

In the third equation, L_{q_{2}} is expressed in terms of L_{q_{0}} and L_{q_{1}}. Hence, if we substitute the third equation into the first equation, then we get

L_{q_{0}} = a · L_{q_{0}} ∪ b · (ϵ ∪ a · L_{q_{1}} ∪ b · L_{q_{0}})
= (a ∪ bb) · L_{q_{0}} ∪ ba · L_{q_{1}} ∪ b.

We obtain the following set of equations.

\{ \begin{matrix}L_{q_{0}} & = & (a ∪ bb) · L_{q_{0}} ∪ ba · L_{q_{1}} ∪ b \\ L_{q_{1}} & = & b · L_{q_{1}} ∪ a · L_{q_{0}} \end{matrix}

Let L = L_{q_{1}}, B = b, and C = a · L_{q_{0}}. Then ϵ ∉ B and L = BL ∪ C. Hence, by Lemma 2.8.2,

L_{q_{1}} = L = B^{∗}C = b^{∗}a · L_{q_{0}}.

If we substitute L_{q_{1}} into the first equation, then we get

L_{q_{0}} = (a ∪ bb) · L_{q_{0}} ∪ bab^{∗}a · L_{q_{0}} ∪ b
= (a ∪ bb ∪ bab^{∗}a)L_{q_{0}} ∪ b.

Again applying Lemma 2.8.2, this time with L = L_{q_{0}}, B = a ∪ bb ∪ bab^{∗}a and C = b, gives

L_{q_{0}} = (a ∪ bb ∪ bab^{∗}a)^{∗} b.

Hence, the regular expression that describes the language accepted by M is

(a ∪ bb ∪ bab^{∗}a)^{∗} b.

2.9 The pumping lemma and nonregular languages

In the previous sections, we have seen that that the class of regular languages is closed under various operations, and that these languages can be described by a (deterministic or nondeterministic) finite automaton, or by a regular expression. These properties helped in developing techniques for showing that a language is regular. In this section, we will present a tool that can be used to prove that certain languages are not regular. Observe that for a regular language,

1. the amount of memory that is needed to determine whether or not a given string is in the language is finite, and

2. if the language consists of an infinite number of strings, then this language should contain infinite subsets with fairly repetitive structures.

Intuitively, languages that do not follow 1. or 2. should be nonregular. For example, consider the language

{0^{n}1^{n} : n ≥ 0}.

This language should be nonregular, because it seems unlikely that a DFA can remember how many 0s it has seen when it has reached the border between the 0s and the 1s while reading a string. Similarly the language

2.9. THE PUMPING LEMMA AND NONREGULAR LANGUAGES 65

{0^{p} : p is a prime number} should be nonregular, because the prime numbers do not have any repetitive structure that can be used by a DFA. To be more rigorous about this, we will establish a property that all regular languages must satisfy. This property is called the pumping lemma. If a language does not have this property, then it must be nonregular.

The pumping lemma states that any sufficiently long string in a regular language can be pumped, i.e., there is a section in that string that can be repeated any number of times, so that the resulting strings are all in the language.

Theorem 2.9.1 (Pumping Lemma for Regular Languages) Let A be a regular language. Then there exists an integer p ≥ 1, called the pumping length, such that the following holds: Every string s in A, with |s| ≥ p, can be written as s = xyz, such that

1. y ≠ ϵ (i.e., |y| ≥ 1),

2. |xy| ≤ p, and

3. xy^{i}z ∈ A, for all i ≥ 0.

In words, the pumping lemma states that by replacing the portion y in s by zero or more copies of it, the resulting string is still in the language A.

Proof. Since A is a regular language, there exists a DFA M = (Q, Σ, δ, q, F) that accepts A. We define p to be the number of states in Q.

Let s = s_{1}s_{2} … s_{n} be an arbitrary string in A, such that n ≥ p. Define r_{1} = q, r_{2} = δ(r_{1}, s_{1}), r_{3} = δ(r_{2}, s_{2}), …, r_{n+1} = δ(r_{n}, s_{n}). Hence, when the DFA M reads the string s, it visits the states r_{1}, r_{2}, …, r_{n+1}. Since s ∈ A, we know that r_{n+1} ∈ F.

Consider the first p + 1 states r_{1}, r_{2}, …, r_{p+1} in this sequence. Since M contains p states, the pigeon hole principle implies that there must be a state that occurs twice in this sequence. That is, there are indices j and ℓ such that 1 ≤ j < ℓ ≤ p + 1 and r_{j} = r_{ℓ}.

q = r_{1}  r_{n+1} r_{j} = r_{ℓ} read x  read y read z

We define x = s_{1}s_{2} … s_{j−1}, y = s_{j} … s_{ℓ−1}, and z = s_{ℓ} … s_{n}. Since j < ℓ, we have y ≠ ϵ, proving the first claim in the theorem. Since ℓ ≤ p + 1, we have |xy| = ℓ − 1 ≤ p, proving the second claim in the theorem. To see that the third claim also holds, recall that the string s = xyz is accepted by M. While reading x, M moves from the start state q to state r_{j}. While reading y, it moves from state r_{j} to state r_{ℓ} = r_{j}; hence, after having read y, M is again in state r_{j}. While reading z, M moves from state r_{j} to the accept state r_{n+1}. Therefore, the substring y can be repeated any number i ≥ 0 of times, and the corresponding string xy^{i}z will still be in A.

2.9.1 Applications of the pumping lemma

First example

Consider the language

A = {0^{n}1^{n} : n ≥ 0}.

We will prove by contradiction that A is not a regular language.

Assume that A is a regular language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Consider the string s = 0^{p}1^{p}. It is clear that s ∈ A and |s| = 2p ≥ p. Hence, by the pumping lemma, s can be written as s = xyz, where y ≠ ϵ, |xy| ≤ p, and xy^{i}z ∈ A for all i ≥ 0.

Observe that, since |xy| ≤ p, the string y contains only 0s. Moreover, since y ≠ ϵ, y contains at least one 0. But now we are in trouble: none of the strings xy^{0}z = xz, xy^{2}z = xyyz, xy^{3}z = xyyyz, …, is contained in A. But, by the pumping lemma, all these strings must be in A. Hence, we have a contradiction and we conclude that A is not a regular language.

Second example

Consider the language

A = {w ∈ {0, 1}^{∗} : the number of 0s in w equals the number of 1s in w}.

Again, we prove by contradiction that A is not a regular language.

Assume that A is a regular language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Consider the string s = 0^{p}1^{p}. Then s ∈ A and |s| = 2p ≥ p. By the pumping lemma, s can be written as s = xyz, where y ≠ ϵ, |xy| ≤ p, and xy^{i}z ∈ A for all i ≥ 0.

Since |xy| ≤ p, the string y contains only 0s. Since y ≠ ϵ, y contains at least one 0. Since the string xy^{2}z = xyyz contains more 0s than 1s, this string is not contained in A. But, by the pumping lemma, this string is contained in A. This is a contradiction and, therefore, A is not a regular language.

Third example

Consider the language

A = {ww : w ∈ {0, 1}^{∗}}.

We prove by contradiction that A is not a regular language.

Assume that A is a regular language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Consider the string s = 0^{p}10^{p}1. Then s ∈ A and |s| = 2p + 2 ≥ p. By the pumping lemma, s can be written as s = xyz, where y ≠ ϵ, |xy| ≤ p, and xy^{i}z ∈ A for all i ≥ 0.

Since |xy| ≤ p, the string y contains only 0s. Since y ≠ ϵ, y contains at least one 0. Therefore, the string xy^{2}z = xyyz is not contained in A. But, by the pumping lemma, this string is contained in A. This is a contradiction and, therefore, A is not a regular language.

Fourth example

Consider the language

A = {0^{m}1^{n} : m > n ≥ 0}.

We prove by contradiction that A is not a regular language.

Assume that A is a regular language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Consider the string s = 0^{p+1}1^{p}. Then s ∈ A and |s| = 2p + 1 ≥ p. By the pumping lemma, s can be written as s = xyz, where y ≠ ϵ, |xy| ≤ p, and xy^{i}z ∈ A for all i ≥ 0.

Since |xy| ≤ p, the string y contains only 0s. Since y ≠ ϵ, y contains at least one 0. Consider the string xy^{0}z = xz. The number of 1s in this string is equal to p, whereas the number of 0s is at most equal to p. Therefore, the string xy^{0}z is not contained in A. But, by the pumping lemma, this string is contained in A. This is a contradiction and, therefore, A is not a regular language.

Fifth example

Consider the language

A = {1^{n^{2}} : n ≥ 0}.

We prove by contradiction that A is not a regular language.

Assume that A is a regular language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Consider the string s = 1^{p^{2}}. Then s ∈ A and |s| = p^{2} ≥ p. By the pumping lemma, s can be written as s = xyz, where y ≠ ϵ, |xy| ≤ p, and xy^{i}z ∈ A for all i ≥ 0.

Observe that

|s| = |xyz| = p^{2}

and

|xy^{2}z| = |xyyz| = |xyz| + |y| = p^{2} + |y|.

Since |xy| ≤ p, we have |y| ≤ p. Since y ≠ ϵ, we have |y| ≥ 1. It follows that

p^{2} < |xy^{2}z| ≤ p^{2} + p < (p + 1)^{2}.

Hence, the length of the string xy^{2}z is strictly between two consecutive squares. In particular, this length is not a square and, therefore, xy^{2}z is not contained in A. But, by the pumping lemma, this string is contained in A. This is a contradiction and, therefore, A is not a regular language.

Sixth example

Consider the language

A = {1^{n} : n is a prime number}.

We prove by contradiction that A is not a regular language.

Assume that A is a regular language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Let n > p be a prime number, and consider the string s = 1^{n}. Then s ∈ A and |s| = n ≥ p. By the pumping lemma, s can be written as s = xyz, where y ≠ ϵ, |xy| ≤ p, and xy^{i}z ∈ A for all i ≥ 0.

Let k be the integer such that y = 1^{k}. Since y ≠ ϵ, we have k ≥ 1. For each i ≥ 0, n + (i − 1)k is a prime number, because xy^{i}z = 1^{n+(i−1)k} ∈ A. For i = n + 1, however, we have

n + (i − 1)k = n + nk = n(1 + k),

which is not a prime number, because n ≥ 2 and 1 + k ≥ 2. This is a contradiction and, therefore, A is not a regular language.

Seventh example

Consider the language

A = {w ∈ {0, 1}^{∗} : the number of occurrences of 01 in w is equal to

the number of occurrences of 10 in w }.

Since this language has the same flavor as the one in the second example, we may suspect that A is not a regular language. This is, however, not true: as we will show, A is a regular language.

The key property is the following one: Let w be an arbitrary string in {0, 1}^{∗}. Then

the absolute value of the number of occurrences of 01 in w minus the number of occurrences of 10 in w is at most one.

This property holds, because between any two consecutive occurrences of 01, there must be exactly one occurrence of 10. Similarly, between any two consecutive occurrences of 10, there must be exactly one occurrence of 01.

We will construct a DFA that accepts A. This DFA uses the following five states:

• q: start state.

• q_{01}: the last symbol read was 1, and the number of occurrences of 01 is one more than the number of occurrences of 10.

70 CHAPTER 2. FINITE AUTOMATA AND REGULAR LANGUAGES

• q_{10}: the last symbol read was 0, and the number of occurrences of 10 is one more than the number of occurrences of 01.

• q_{equal}^{0}: the last symbol read was 0, and the number of occurrences of 01 is equal to the number of occurrences of 10.

• q_{equal}^{1}: the last symbol read was 1, and the number of occurrences of 01

is equal to the number of occurrences of 10. The set of accept states is equal to {q, q_{equal}^{0}, q_{equal}^{1}}. The state diagram of the DFA is given below.

q_{equal}^{0} q_{equal}^{1}  q_{01} q_{10} q  0  0  1  1 0 1  0  0 1 1

2.10 Exercises

2.1 For each of the following languages, construct a DFA that accepts the language. In all cases, the alphabet is {0, 1}.

1. {w : w is a binary string}

2. {w : the length of w is divisible by three}

3. {w : 110 is not a substring of w} 2.10. EXERCISES 71

4. {w : w contains at least five 1s}

5. {w : w contains the substring 1011}

6. {w : w contains at least two 1s and at most two 0s}

7. {w : w contains an odd number of 1s or exactly two 0s}

8. {w : w contains even number of 0s and even number of 1s} 2.2 Construct two NFAs M_{1} and M_{2}, different from those in these notes, both consisting of at least three states and both containing at least one cycle, over the alphabet {0, 1}. Answer the following questions with some reasonable justification:

1. Why are M_{1} and M_{2} NFAs?

2. Describe the languages L(M_{1}) and L(M_{2}), that is, describe the set of strings that are accepted by M_{1} and M_{2}.

3. Describe DFAs N_{1} and N_{2}, such that L(N_{1}) = L(M_{1}) and L(N_{2}) = L(M_{2}). Argue why this is true.

4. Construct an NFA for L(M_{1}) ∪ L(M_{2}).

5. Construct an NFA for L(M_{1})L(M_{2}).

6. Construct NFAs for (L(M_{1}))^{∗} and (L(M_{2}))^{∗}.

7. Construct a regular expression that describes L(M_{1}). 2.3 For each of the following languages, construct an NFA, with the specified number of states, that accepts the language. In all cases, the alphabet is {0, 1}.

1. The language {w : w ends with 10} with three states.

2. The language {w : w contains the substring 1011} with five states.

3. The language {w : w contains an odd number of 1s or exactly two 0s} with six states.

72 CHAPTER 2. FINITE AUTOMATA AND REGULAR LANGUAGES 2.4 Give regular expressions describing the following languages. In all cases, the alphabet is {0, 1}.

1. {w : w contains at least three 1s}.

2. {w : w contains at least two 1s and at most one 0},

3. {w : w contains an even number of 0s and exactly two 1s}. 2.5 For each of the following languages, give two strings that are members and two strings that are not members, a total of four strings each. The alphabet is Σ = {a, b}.

1. a(ba)^{∗}b.

2. Σ^{∗}aΣ^{∗}bΣ^{∗}aΣ^{∗}.

3. (a ∪ ba ∪ bb)Σ^{∗} 2.6 Convert the following NFA to an equivalent DFA.

2 ϵ, b  1 a a a  3  b

2.7 Convert each of the following regular expressions to an NFA. 1. (0 ∪ 1)^{∗}000(0 ∪ 1)^{∗}

2. (((10)^{∗}(00)) ∪ 10)^{∗}

2.8 Convert the following DFA to a regular expression.

1  2 3 a  a b b  a b

2.9 Let A be a regular language. Prove that there exists an NFA that accepts A and that has exactly one accept state.

2.10 For any string w = w_{1}w_{2} … w_{n}, we denote by w^{R} the string obtained by reading w backwards, i.e., w^{R} = w_{n}w_{n−1} … w_{2}w_{1}. For any language A, we define A^{R} to be the language obtained by reading all strings in A backwards, i.e.,

A^{R} = {w^{R} : w ∈ A}.

Let A be a regular language. Prove that the language A^{R} is also regular.

2.11 Let Σ be a non-empty alphabet, and let L be a language over Σ, i.e., L ⊆ Σ^{∗}. We define a binary relation R_{L} on Σ^{∗} × Σ^{∗}, in the following way: For any two strings u and u′ in Σ^{∗},

uR_{L}u′ if and only if (∀v ∈ Σ^{∗} : uv ∈ L ⇔ u′v ∈ L) .

Prove that R_{L} is an equivalence relation.

2.12 Let Σ = {0, 1}, let

L = {w ∈ Σ^{∗} : |w| is odd},

and consider the relation R_{L} defined in the previous exercise.

1. Prove that for any two strings u and u′ in Σ^{∗},

uR_{L}u′ ⇔ |u| − |u′| is even.

2. Determine all equivalence classes of the relation R_{L}.

2.13 Let Σ be a non-empty alphabet, and let L be a language over Σ, i.e., L ⊆ Σ^{∗}. Recall the equivalence relation R_{L} that was defined above.

1. Assume that L is a regular language, and let M = (Q, Σ, δ, q_{0}, F) be a DFA that accepts L. Let u and u′ be strings in Σ^{∗}. Let q be the state reached, when following the path in the state diagram of M, that starts in q_{0} and that is obtained by reading the string u. Similarly, let q′ be the state reached, when following the path in the state diagram of M, that starts in q_{0} and that is obtained by reading the string u′.

Prove the following: If q = q′, then uR_{L}u′.

2. Prove the following claim: If L is a regular language, then the equivalence relation R_{L} has a finite number of equivalence classes.

2.14 Let L be the language defined by

L = {uu^{R} : u ∈ {0, 1}^{∗}}.

In words, a string is in L if and only if its length is even, and the second half is the reverse of the first half.

1. Let m and n be two distinct positive integers, and consider the two strings u = 0^{m}1 and u′ = 0^{n}1. Prove that ¬(uR_{L}u′).

2. Prove that L is not a regular language, without using the pumping lemma).

2.15 Using the pumping lemma, prove that the following languages are not regular.

1. {a^{n}b^{m}c^{n+m} : n ≥ 0, m ≥ 0}.

2. {a^{n}b^{m}a^{n} : n ≥ 0, m ≥ 0}.

3. {a^{2^{n}} : n ≥ 0}. (Remark: a^{2^{n}} is the string consisting of 2^{n} many a’s.)

2.16 Let L ⊆ Σ^{∗} be a regular language, where Σ = {0, 1}. Let w be any binary string. Show that the language A = {x ∈ L : w is a substring of x} is a regular language.

2.17 Let A be a language over an alphabet Σ = {0, 1}, and define the halflanguage H of A to be set of all first halves of strings in A, i.e., H = {x ∈ Σ^{∗} : for some y ∈ Σ^{∗}, |x| = |y| and xy ∈ A}. Prove that if A is regular then H is also regular.

2.18 In many examples of conversion of an NFA N to an equivalent DFA M using Theorem 2.5.1, we have seen that many states in M are not reachable from the start state. We remove all such “unreachable” states and obtain a reduced equivalent DFA M′. Suggest an efficient algorithm that can remove all such unreachable states in M to obtain M′.

2.19 We are given seven quarter coins, where each coin has two sides: the Queen (=Heads) and the Moose (=Tails). Initially they are placed on a table with the Queen side facing up for all of them. At any moment we can select any four coins and flip them over. Show that we can never reach the configuration where all the coins have the Moose side facing up! (Hint: You can define a state to represent certain number of Queens (say q) and Moose (say m), such that m + q = 7. A transition corresponds to flipping 4 coins.)

2.20 Show that if L is a regular language then the following language is regular as well: {xy|yx ∈ L}.

2.21 Show that if L is regular than so is L_{\frac{1}{3}} , where L_{\frac{1}{3}} = \{c|∃a, b such that, |a| = |b| = |c|, abc ∈ L}.

2.22 Show that if L is regular than so is the SQRT (L), where SQRT (L) = {a|∃b, such that, |b| = |a|^{2}, ab ∈ L}.

2.23 Show that if L is regular than so is {x|xx^{R} ∈ L}.

2.24 Prove or Disprove the following:

1. If L is regular than so is L′, where L ⊆ L′.

2. If L is regular than so is L′, where L′ ⊆ L.

3. If L and LL′ are regular, than L′ is regular.

2.25 Show that every regular language is accepted by a planar NFA. A NFA is planar if it can be drawn in the plane so that no two edges cross accept that they can meet at a state. (See the definition of planar graphs from any Graph Theory book). What about DFA’s? Does there always exist a planar DFA for the corresponding regular language?

Context-Free Languages

In this chapter, we introduce the class of context-free languages. As we will see, this class contains all regular languages. Moreover, this class contains languages such as {0^{n}1^{n} : n ≥ 0}, which, as we have seen in Section 2.9.1, is not a regular language.

The class of context-free languages consists of languages that have some sort of recursive structure. We will see two equivalent methods to obtain this class. We start with context-free grammars, which is a technique used for defining the syntax of programming languages and their compilation. Then we introduce the notion of (nondeterministic) pushdown automata, and show that these automata have the same power as context-free grammars.

3.1 Context-free grammars

We start with an example. Consider the following three (substitution) rules:

\begin{aligned}A & → & 0A1, \\ A & → & B, \\ B & → & $. \end{aligned}

Here, A and B are variables, A is the start variable, and 0, 1, and $ are terminals. We use these rules to derive strings consisting of terminals (i.e., elements of {0, 1, $}^{∗}), in the following manner:

1. Initialize the current string to be the string consisting of the start variable A only.

2. Take a variable in the current string, and take a rule that has this variable on the left-hand side. Then, in the current string, replace this variable by the right-hand side of the rule.

3. Repeat 2. until the current string contains only terminals.

For example, the string 0000$1111 can be derived in the following way:

A ⇒ 0A1
⇒ 00A11
⇒ 000A111
⇒ 0000A1111
⇒ 0000B1111
⇒ 0000$1111

This derivation can also be represented using a parse tree, as in the figure below:

A A A A B $ 0 000  11 11

The three rules in this example constitute a context-free grammar. The language of this grammar is the set of all strings that can be derived from the start variable, and that contain only terminals. For this example, the language is

{0^{n}$1^{n} : n ≥ 0}.

Definition 3.1.1 A context-free grammar is a 4-tuple G = (V, Σ, R, S), where

1. V is a finite set, whose elements are called variables, 2. Σ is a finite set, whose elements are called terminals,

3. V ∩ Σ = ∅,

4. S is an element of V; it is called the start variable,

5. R is a finite set, whose elements are called rules. Each rule has the form A → w, where A ∈ V and w ∈ (V ∪ Σ)^{∗}.

In our example, we have V = {A, B}, Σ = {0, 1, $}, S = A, and

R = {A → 0A1, A → B, B → $}.

Definition 3.1.2 Let G = (V, Σ, R, S) be a context-free grammar. Let u, v, and w be strings in (V ∪ Σ)^{∗}, and let A → w be a rule in R. We say that the string uwv can be derived in one step from the string uAv, and write this as

uAv ⇒ uwv.

In other words, by applying the rule A → w to the string uAv, we obtain the string uwv.

Definition 3.1.3 Let G = (V, Σ, R, S) be a context-free grammar. Let u and v be strings in (V ∪ Σ)^{∗}. We say that v can be derived from u, and write this as u ⇒^{∗} v, if one of the following three conditions holds:

1. u = v, or

2. u ⇒ v, or

3. there exists an integer k ≥ 1, and there exists a sequence u_{1}, u_{2}, …, u_{k} of strings in (V ∪ Σ)^{∗}, such that

u ⇒ u_{1} ⇒ u_{2} ⇒ … ⇒ u_{k} ⇒ v.

In other words, by starting with the string u, and applying rules zero or more times, we obtain the string v.

Definition 3.1.4 Let G = (V, Σ, R, S) be a context-free grammar. The language of G is defined to be the set of all strings in Σ^{∗} that can be derived from the start variable S:

L(G) = {w ∈ Σ^{∗} : S ⇒^{∗} w}.

Definition 3.1.5 A language L is called context-free, if there exists a contextfree grammar G such that L(G) = L.

3.2 Examples of context-free grammars

3.2.1 Properly nested parentheses

Consider the context-free grammar G = (V, Σ, R, S), where V = {S}, Σ = {a, b}, and

R = {S → aSb, S → SS, S → ϵ}.

We write the three rules in R as

S → aSb|SS|ϵ,

where you can think of “|” as being a short-hand for “or”.

For example, by applying the rules in R, starting with the start variable S, we obtain

S ⇒ SS
⇒ aSbS
⇒ aSbSS
⇒ aSSbSS
⇒ aaSbSbSS
⇒ aabSbSS
⇒ aabbSS
⇒ aabbaSbS
⇒ aabbabS
⇒ aabbabaSb
⇒ aabbabab

What is the language L(G) of this context-free grammar G? If we think of a as being a left-parenthesis “(”, and of b as being a right-parenthesis “)”, then L(G) is the language consisting of all strings of properly nested parentheses.

3.2.2 A context-free grammar for a nonregular language

Consider the language L_{1} = {0^{n}1^{n} : n ≥ 0}. We have seen in Section 2.9.1 that L_{1} is not a regular language. We claim that L_{1} is a context-free language.

In order to prove this claim, we have to construct a context-free grammar G_{1} such that L(G_{1}) = L_{1}.

Let G_{1} = (V_{1}, Σ, R_{1}, S_{1}), where V_{1} = {S_{1}}, Σ = {0, 1}, and R_{1} consists of the rules

S_{1} → 0S_{1}1|ϵ.

Hence, R_{1} = {S_{1} → 0S_{1}1, S_{1} → ϵ}. It is not difficult to see that L(G_{1}) = L_{1}.

In a symmetric way, we see that the context-free grammar G_{2} = (V_{2}, Σ, R_{2}, S_{2}), where V_{2} = {S_{2}}, Σ = {0, 1}, and R_{2} consists of the rules

S_{2} → 1S_{2}0|ϵ,

has the property that L(G_{2}) = L_{2}, where L_{2} = {1^{n}0^{n} : n ≥ 0}.

Define L = L_{1} ∪ L_{2}, i.e.,

L = {0^{n}1^{n} : n ≥ 0} ∪ {1^{n}0^{n} : n ≥ 0}.

The context-free grammar G = (V, Σ, R, S), where V = {S, S_{1}, S_{2}}, Σ = {0, 1}, and R consists of the rules

\begin{aligned}S & → & S_{1}|S_{2} \\ S_{1} & → & 0S_{1}1|ϵ \\ S_{2} & → & 1S_{2}0|ϵ, \end{aligned}

has the property that L(G) = L. Hence, L is a context-free language.

3.2.3 A context-free grammar for the complement of a nonregular language

Let L be the (nonregular) language L = {0^{n}1^{n} : n ≥ 0}. We want to prove that the complement \overline{L} of L is a context-free language. Hence, we want to construct a context-free grammar G whose language is equal to \overline{L}. Observe that a binary string w is in \overline{L} if and only if

1. w = 0^{m}1^{n}, for some integers m and n with 0 ≤ m < n, or

2. w = 0^{m}1^{n}, for some integers m and n with 0 ≤ n < m, or

3. w contains 10 as a substring.

Let G = (V, Σ, R, S), where V = {S, T_{1}, T_{2}, T_{3}, X}, Σ = {0, 1}, and R consists of the rules

\begin{aligned}S & → & T_{1}|T_{2}|T_{3} \\ T_{1} & → & 1|T_{1}1|0T_{1}1 \\ T_{2} & → & 0|0T_{2}|0T_{2}1 \\ T_{3} & → & X10X \\ X & → & ϵ|0X|1X \end{aligned}

Observe that

T_{1} ⇒^{∗} 0^{m}1^{n}, for all integers m and n with 0 ≤ m < n,

and

T_{2} ⇒^{∗} 0^{m}1^{n}, for all integers m and n with 0 ≤ n < m.

Next, observe that

X ⇒^{∗} u, for each string u in {0, 1}^{∗},

which implies that

T_{3} ⇒^{∗} w, for every binary string w that contains 10 as a substring.

From these observations, it can be seen that L(G) = \overline{L}.

3.2.4 A context-free grammar that verifies addition

Consider the language

L = {a^{n}b^{m}c^{n+m} : n ≥ 0, m ≥ 0}.

Using the pumping lemma for regular languages (Theorem 2.9.1), it can be shown that L is not a regular language. We will construct a contextfree grammar G whose language is equal to L, thereby proving that L is a context-free language.

First observe that ϵ ∈ L. Therefore, we will take S → ϵ to be one of the rules in the grammar.

Let us see how we can derive all strings in L from the start variable S:

1. Every time we add an a, we also add a c. In this way, we obtain all strings of the form a^{n}c^{n}, where n ≥ 0.

2. Given a string of the form a^{n}c^{n}, we start adding bs. Every time we add a b, we also add a c. Observe that every b has to be added between the as and the cs. Therefore, we use a variable B as a “pointer” to the position in the current string where a b can be added.

We obtain the context-free grammar G = (V, Σ, R, S), where V = {S, A, B}, Σ = {a, b, c}, and R consists of the rules

\begin{aligned}S & → & ϵ|A \\ A & → & ϵ|aAc|B \\ B & → & ϵ|bBc \end{aligned}

The facts that

• A ⇒^{∗} a^{n}c^{n}, for every n ≥ 0,

• A ⇒^{∗} a^{n}Bc^{n}, for every n ≥ 0,

• B ⇒^{∗} b^{m}c^{m}, for every m ≥ 0,

imply that the following strings can be derived from the start variable S:

• S ⇒^{∗} a^{n}c^{n}, for every n ≥ 0,

• S ⇒^{∗} a^{n}Bc^{n} ⇒^{∗} a^{n}b^{m}c^{m}c^{n} = a^{n}b^{m}c^{n+m}, for all n ≥ 0 and m ≥ 0.

That is, we have L(G) = L. In fact, since

S ⇒ A ⇒ B ⇒ ϵ,

we can simplify this grammar G, by eliminating the rules S → ϵ and A → ϵ. This gives the context-free grammar G′ = (V, Σ, R′, S), where V = {S, A, B}, Σ = {a, b, c}, and R′ consists of the rules

\begin{aligned}S & → & A \\ A & → & aAc|B \\ B & → & ϵ|bBc \end{aligned}

Finally, observe that we do not need S but, instead, can use A as start variable. This gives our final context-free grammar G′′ = (V, Σ, R′′, S), where V = {S, A, B}, Σ = {a, b, c}, and R′′ consists of the rules

\begin{aligned}A & → & aAc|B \\ B & → & ϵ|bBc \end{aligned}

3.3 Regular languages are context-free

We mentioned already that the class of context-free languages includes the class of regular languages. In this section, we will prove this claim.

Theorem 3.3.1 Let Σ be an alphabet, and let L ⊆ Σ^{∗} be a regular language. Then L is a context-free language.

Proof. Since L is a regular language, there exists a deterministic finite automaton M = (Q, Σ, δ, q, F) that accepts L. Define the context-free grammar G = (V, Σ, R, S), where

• V = Q; hence, the variables of G are the states of M,

• S = q; hence the start variable of G is the start state of M, and

• R consists of the rules

A → aB, where A ∈ Q, a ∈ Σ, B ∈ Q, and δ(A, a) = B,

and

A → ϵ, where A ∈ F.

In words,

• every transition δ(A, a) = B of M (i.e., when M is in the state A and reads the symbol a, it switches to the state B) corresponds to a rule A → aB in the grammar G,

• every accept state A of M corresponds to a rule A → ϵ in the grammar G.

We claim that L(G) = L. In order to prove this, we have to show that L(G) ⊆ L and L ⊆ L(G).

We prove that L ⊆ L(G). Let w = w_{1}w_{2} … w_{n} be an arbitrary string in L. When the finite automaton M reads the string w, it visits the states r_{0}, r_{1}, …, r_{n}, where

• r_{0} = q, and

• r_{i+1} = δ(r_{i}, w_{i+1}) for i = 0, 1, …, n − 1.

Since w ∈ L = L(M), we know that r_{n} ∈ F.

It follows from the way we defined the grammar G that

• for each i = 0, 1, …, n − 1, r_{i} → w_{i+1}r_{i+1} is a rule in R, and

• r_{n} → ϵ is a rule in R.

Therefore, we have

S = q = r_{0} ⇒ w_{1}r_{1} ⇒ w_{1}w_{2}r_{2} ⇒ … ⇒ w_{1}w_{2} … w_{n}r_{n} ⇒ w_{1}w_{2} … w_{n} = w.

This proves that w ∈ L(G).

The proof of the claim that L(G) ⊆ L is left as an exercise.

In Sections 2.9.1 and 3.2.2, we have seen that the language {0^{n}1^{n} : n ≥ 0} is not regular, but context-free. Therefore, the class of all context-free languages properly contains the class of regular languages.

3.3.1 An example

Let L be the language defined as

L = {w ∈ {0, 1}^{∗} : 101 is a substring of w}.

In Section 2.2.2, we have seen that L is a regular language. In that section, we constructed the following deterministic finite automaton M that accepts L (we have renamed the states):

0  1  1 0  0  1  0, 1 S  A B  C

We apply the construction given in the proof of Theorem 3.3.1 to convert M to a context-free grammar G whose language is equal to L. According to this construction, we have G = (V, Σ, R, S), where V = {S, A, B, C}, Σ = {0, 1}, the start variable S is the start state of M, and R consists of the rules

\begin{aligned}S & → & 0S|1A \\ A & → & 0B|1A \\ B & → & 0S|1C \\ C & → & 0C|1C|ϵ \end{aligned}

Consider the string 010011011, which is an element of L. When the finite automaton M reads this string, it visits the states

S, S, A, B, S, A, A, B, C, C.

In the grammar G, this corresponds to the derivation

S ⇒ 0S
⇒ 01A
⇒ 010B
⇒ 0100S
⇒ 01001A
⇒ 010011A
⇒ 0100110B
⇒ 01001101C
⇒ 010011011C
⇒ 010011011.

Hence,

S ⇒^{∗} 010011011.

3.4 Chomsky normal form

The rules in a context-free grammar G = (V, Σ, R, S) are of the form

A → w,

where A is a variable and w is a string over the alphabet V ∪ Σ. In this section, we show that every context-free grammar G can be converted to a context-free grammar G′, such that L(G) = L(G′), and the rules of G′ are of a restricted form, as specified in the following definition:

Definition 3.4.1 A context-free grammar G = (V, Σ, R, S) is said to be in Chomsky normal form, if every rule in R has one of the following three forms:

1. A → BC, where A, B, and C are elements of V, B ≠ S, and C ≠ S.

2. A → a, where A is an element of V, and a is an element of Σ.

3. S → ϵ, where S is the start variable.

Theorem 3.4.2 Let Σ be an alphabet, and let L be a context-free language. There exists a context-free grammar in Chomsky normal form, whose language is L.

Proof. Since L is a context-free language, there exists a context-free grammar G = (V, Σ, R, S), such that L(G) = L. We will transform G into a grammar that is in Chomsky normal form and whose language is equal to L(G). The transformation consists of five steps.

Step 1: Eliminate the start variable from the right-hand side of the rules.

We define G_{1} = (V_{1}, Σ, R_{1}, S_{1}), where S_{1} is the start variable (which is a new variable), V_{1} = V ∪ {S_{1}}, and R_{1} = R ∪ {S_{1} → S}. This grammar has the property that

• the start variable S_{1} does not occur on the right-hand side of any rule in R_{1}, and

• L(G_{1}) = L(G).

Step 2: An ϵ-rule is a rule that is of the form A → ϵ, where A is a variable that is not equal to the start variable. In the second step, we eliminate all ϵ-rules from G_{1}.

We consider all ϵ-rules, one after another. Let A → ϵ be one such rule, where A ∈ V_{1} and A ≠ S_{1}. We modify G_{1} as follows:

1. Remove the rule A → ϵ from the current set R_{1}.

2. For each rule in the current set R_{1} that is of the form

(a) B → A, add the rule B → ϵ to R_{1}, unless this rule has already been deleted from R_{1}; observe that in this way, we replace the twostep derivation B ⇒ A ⇒ ϵ by the one-step derivation B → ϵ; (b) B → uAv (where u and v are strings that are not both empty), add the rule B → uv to R_{1}; observe that in this way, we replace the two-step derivation B ⇒ uAv ⇒ uv by the one-step derivation B → uv;

(c) B → uAvAw (where u, v, and w are strings), add the rules B → uvw, B → uAvw, and B → uvAw to R_{1}; if u = v = w = ϵ and the rule B → ϵ has already been deleted from R_{1}, then we do not add the rule B → ϵ);

(d) treat rules in which A occurs more than twice on the right-hand side in a similar fashion.

We repeat this process until all ϵ-rules have been eliminated. Let R_{2} be the set of rules, after all ϵ-rules have been eliminated. We define G_{2} = (V_{2}, Σ, R_{2}, S_{2}), where V_{2} = V_{1} and S_{2} = S_{1}. This grammar has the property that

• the start variable S_{2} does not occur on the right-hand side of any rule in R_{2},

• R_{2} does not contain any ϵ-rule (it may contain the rule S_{2} → ϵ), and

• L(G_{2}) = L(G_{1}) = L(G).

Step 3: A unit-rule is a rule that is of the form A → B, where A and B are variables. In the third step, we eliminate all unit-rules from G_{2}.

We consider all unit-rules, one after another. Let A → B be one such rule, where A and B are elements of V_{2}. We know that B ≠ S_{2}. We modify G_{2} as follows:

1. Remove the rule A → B from the current set R_{2}.

2. For each rule in the current set R_{2} that is of the form B → u, where u ∈ (V_{2} ∪ Σ)^{∗}, add the rule A → u to the current set R_{2}, unless this is a unit-rule that has already been eliminated. Observe that in this way, we replace the two-step derivation A ⇒ B ⇒ u by the one-step derivation A ⇒ u.

We repeat this process until all unit-rules have been eliminated. Let R_{3} be the set of rules, after all unit-rules have been eliminated. We define G_{3} = (V_{3}, Σ, R_{3}, S_{3}), where V_{3} = V_{2} and S_{3} = S_{2}. This grammar has the property that • the start variable S_{3} does not occur on the right-hand side of any rule

in R_{3},

• R_{3} does not contain any ϵ-rule (it may contain the rule S_{3} → ϵ),

• R_{3} does not contain any unit-rule, and

• L(G_{3}) = L(G_{2}) = L(G_{1}) = L(G).

Step 4: Eliminate all rules having more than two symbols on the right-hand side.

For each rule in the current set R_{3} that is of the form A → u_{1}u_{2} … u_{k}, where k ≥ 3 and each u_{i} is an element of V_{3} ∪ Σ, we modify G_{3} as follows:

1. Remove the rule A → u_{1}u_{2} … u_{k} from the current set R_{3}.

2. Add the following rules to the current set R_{3}:

A → u_{1}A_{1}, A_{1} → u_{2}A_{2}, A_{2} → u_{3}A_{3}, …, A_{k−2} → u_{k−1}u_{k},

where A_{1}, A_{2}, …, A_{k−2} are new variables that are added to the current set V_{3}.

Observe that in this way, we replace the one-step derivation A ⇒ u_{1}u_{2} … u_{k} by the (k − 1)-step derivation

A ⇒ u_{1}A_{1} ⇒ u_{1}u_{2}A_{2} ⇒ … ⇒ u_{1}u_{2} … u_{k−2}A_{k−2} ⇒ u_{1}u_{2} … u_{k}.

Let R_{4} be the set of rules, and let V_{4} be the set of variables, after all rules with more than two symbols on the right-hand side have been eliminated. We define G_{4} = (V_{4}, Σ, R_{4}, S_{4}), where S_{4} = S_{3}. This grammar has the property that

• the start variable S_{4} does not occur on the right-hand side of any rule in R_{4},

• R_{4} does not contain any ϵ-rule (it may contain the rule S_{4} → ϵ),

• R_{4} does not contain any unit-rule,

• R_{4} does not contain any rule with more than two symbols on the righthand side, and

• L(G_{4}) = L(G_{3}) = L(G_{2}) = L(G_{1}) = L(G).

Step 5: Eliminate all rules of the form A → u_{1}u_{2}, where u_{1} and u_{2} are not both variables.

For each rule in the current set R_{4} that is of the form A → u_{1}u_{2}, where u_{1} and u_{2} are elements of V_{4} ∪ Σ, but u_{1} and u_{2} are not both contained in V_{4}, we modify G_{3} as follows:

1. If u_{1} ∈ Σ and u_{2} ∈ V_{4}, then replace the rule A → u_{1}u_{2} in the current set R_{4} by the two rules A → U_{1}u_{2} and U_{1} → u_{1}, where U_{1} is a new variable that is added to the current set V_{4}.

Observe that in this way, we replace the one-step derivation A ⇒ u_{1}u_{2} by the two-step derivation A ⇒ U_{1}u_{2} ⇒ u_{1}u_{2}.

2. If u_{1} ∈ V_{4} and u_{2} ∈ Σ, then replace the rule A → u_{1}u_{2} in the current set R_{4} by the two rules A → u_{1}U_{2} and U_{2} → u_{2}, where U_{2} is a new variable that is added to the current set V_{4}.

Observe that in this way, we replace the one-step derivation A ⇒ u_{1}u_{2} by the two-step derivation A ⇒ u_{1}U_{2} ⇒ u_{1}u_{2}.

3. If u_{1} ∈ Σ and u_{2} ∈ Σ, then replace the rule A → u_{1}u_{2} in the current set R_{4} by the three rules A → U_{1}U_{2}, U_{1} → u_{1}, and U_{2} → u_{2}, where U_{1} and U_{2} are new variables that are added to the current set V_{4}.

Observe that in this way, we replace the one-step derivation A ⇒ u_{1}u_{2} by the three-step derivation A ⇒ U_{1}U_{2} ⇒ u_{1}U_{2} ⇒ u_{1}u_{2}.

Let R_{5} be the set of rules, and let V_{5} be the set of variables, after Step 5 has been completed. We define G_{5} = (V_{5}, Σ, R_{5}, S_{5}), where S_{5} = S_{4}. This grammar has the property that

• the start variable S_{5} does not occur on the right-hand side of any rule in R_{5},

• R_{5} does not contain any ϵ-rule (it may contain the rule S_{5} → ϵ),

• R_{5} does not contain any unit-rule,

• R_{5} does not contain any rule with more than two symbols on the righthand side, • R_{5} does not contain any rule of the form A → u_{1}u_{2}, where u_{1} and u_{2}

are not both variables of V_{5}, and

• L(G_{5}) = L(G_{4}) = L(G_{3}) = L(G_{2}) = L(G_{1}) = L(G).

Since the grammar G_{5} is in Chomsky normal form, the proof is complete.

3.4.1 An example

Consider the context-free grammar G = (V, Σ, R, A), where V = {A, B}, Σ = {0}, A is the start variable, and R consists of the rules

\begin{aligned}A & → & BAB|B|ϵ \\ B & → & 00|ϵ \end{aligned}

We apply the construction given in the proof of Theorem 3.4.2 to convert this grammar to a context-free grammar in Chomsky normal form whose language is the same as that of G. Throughout the construction, upper case letters will denote variables.

Step 1: Eliminate the start variable from the right-hand side of the rules.

We introduce a new start variable S, and add the rule S → A. This gives the following grammar:

\begin{aligned}S & → & A \\ A & → & BAB|B|ϵ \\ B & → & 00|ϵ \end{aligned}

Step 2: Eliminate all ϵ-rules.

We take the ϵ-rule A → ϵ, and remove it. Then we consider all rules that contain A on the right-hand side. There are two such rules:

• S → A; we add the rule S → ϵ;

• A → BAB; we add the rule A → BB.

This gives the following grammar:

\begin{aligned}S & → & A|ϵ \\ A & → & BAB|B|BB \\ B & → & 00|ϵ \end{aligned}

We take the ϵ-rule B → ϵ, and remove it. Then we consider all rules that contain B on the right-hand side. There are three such rules:

• A → BAB; we add the rules A → AB, A → BA, and A → A;

• A → B; we do not add the rule A → ϵ, because it has already been removed;

• A → BB; we add the rule A → B, but not the rule A → ϵ (because it has already been removed).

At this moment, we have the following grammar:

\begin{aligned}S & → & A|ϵ \\ A & → & BAB|B|BB|AB|BA|A \\ B & → & 00 \end{aligned}

Since all ϵ-rules have been eliminated, this completes Step 2. (Observe that the rule S → ϵ is allowed, because S is the start variable.)

Step 3: Eliminate all unit-rules.

We take the unit-rule A → A. We can remove this rule, without adding any new rule. At this moment, we have the following grammar:

\begin{aligned}S & → & A|ϵ \\ A & → & BAB|B|BB|AB|BA \\ B & → & 00 \end{aligned}

We take the unit-rule S → A, remove it, and add the rules

S → BAB|B|BB|AB|BA.

This gives the following grammar:

\begin{aligned}S & → & ϵ|BAB|B|BB|AB|BA \\ A & → & BAB|B|BB|AB|BA \\ B & → & 00 \end{aligned}

We take the unit-rule S → B, remove it, and add the rule S → 00. This gives the following grammar:

\begin{aligned}S & → & ϵ|BAB|BB|AB|BA|00 \\ A & → & BAB|B|BB|AB|BA \\ B & → & 00 \end{aligned}

We take the unit-rule A → B, remove it, and add the rule A → 00. This gives the following grammar:

\begin{aligned}S & → & ϵ|BAB|BB|AB|BA|00 \\ A & → & BAB|BB|AB|BA|00 \\ B & → & 00 \end{aligned}

Since all unit-rules have been eliminated, this concludes Step 3.

Step 4: Eliminate all rules having more than two symbols on the right-hand side. There are two such rules:

• We take the rule S → BAB, remove it, and add the rules S → BA_{1} and A_{1} → AB.

• We take the rule A → BAB, remove it, and add the rules A → BA_{2}

and A_{2} → AB. This gives the following grammar:

\begin{aligned}S & → & ϵ|BB|AB|BA|00|BA_{1} \\ A & → & BB|AB|BA|00|BA_{2} \\ B & → & 00 \\ A_{1} & → & AB \\ A_{2} & → & AB \end{aligned}

Step 4 is completed now.

Step 5: Eliminate all rules, whose right-hand side contains exactly two symbols, which are not both variables. There are three such rules:

• We replace the rule S → 00 by the rules S → A_{3}A_{3} and A_{3} → 0.

• We replace the rule A → 00 by the rules A → A_{4}A_{4} and A_{4} → 0.

• We replace the rule B → 00 by the rules B → A_{5}A_{5} and A_{5} → 0. This gives the following grammar, which is in Chomsky normal form:

\begin{aligned}S & → & ϵ|BB|AB|BA|BA_{1}|A_{3}A_{3} \\ A & → & BB|AB|BA|BA_{2}|A_{4}A_{4} \\ B & → & A_{5}A_{5} \\ A_{1} & → & AB \\ A_{2} & → & AB \\ A_{3} & → & 0 \\ A_{4} & → & 0 \\ A_{5} & → & 0 \end{aligned}

3.5 Pushdown automata

In this section, we introduce nondeterministic pushdown automata. As we will see, the class of languages that can be accepted by these automata is exactly the class of context-free languages.

We start with an informal description of a deterministic pushdown automaton. Such an automaton consists of the following, see also Figure 3.1.

1. There is a tape which is divided into cells. Each cell stores a symbol belonging to a finite set Σ, called the tape alphabet. There is a special symbol □ that is not contained in Σ; this symbol is called the blank symbol. If a cell contains □, then this means that the cell is actually empty.

2. There is a tape head which can move along the tape, one cell to the right per move. This tape head can also read the cell it currently scans.

3. There is a stack containing symbols from a finite set Γ, called the stack alphabet. This set contains a special symbol $.

4. There is a stack head which can read the top symbol of the stack. This head can also pop the top symbol, and it can push symbols of Γ onto the stack.

5. There is a state control, which can be in any one of a finite number of states. The set of states is denoted by Q. The set Q contains one special state q, called the start state.

The input for a pushdown automaton is a string in Σ^{∗}. This input string is stored on the tape of the pushdown automaton and, initially, the tape head is on the leftmost symbol of the input string. Initially, the stack contains only the special symbol $, and the pushdown automaton is in the start state q. In one computation step, the pushdown automaton does the following:

1. Let us say that the pushdown automaton is currently in state r. Let a be the symbol of Σ that is read by the tape head, and let A be the symbol of Γ that is on top of the stack.

2. Depending on the current state r, the tape symbol a, and the stack symbol A, state control

a a b a b b a b a b □ tape

$ A A B A

stack

Figure 3.1: A pushdown automaton.

(a) the pushdown automaton switches to a state r′ of Q (which may be equal to r),

(b) the tape head either moves one cell to the right or stays at the current cell, and

(c) the top symbol A is replaced by a string w that belongs to Γ^{∗}. To be more precise,

i. if w = ϵ, then A is popped from the stack, whereas ii. if w = B_{1}B_{2} … B_{k}, with k ≥ 1 and B_{1}, B_{2}, …, B_{k} ∈ Γ, then A is replaced by w, and B_{k} becomes the new top symbol of the stack.

Later, we will specify when the pushdown automaton accepts the input string.

We now give a formal definition of a deterministic pushdown automaton.

Definition 3.5.1 A deterministic pushdown automaton is a 5-tuple M = (Σ, Γ, Q, δ, q), where 96 CHAPTER 3. CONTEXT-FREE LANGUAGES

1. Σ is a finite set, called the tape alphabet; the blank symbol □ is not contained in Σ,

2. Γ is a finite set, called the stack alphabet; this alphabet contains the special symbol $,

3. Q is a finite set, whose elements are called states,

4. q is an element of Q; it is called the start state,

5. δ is called the transition function, which is a function δ : Q × (Σ ∪ {□}) × Γ → Q × {N, R} × Γ^{∗}.

The transition function δ can be thought of as being the “program” of the pushdown automaton. This function tells us what the automaton can do in “one computation step”: Let r ∈ Q, a ∈ Σ ∪ {□}, and A ∈ Γ. Furthermore, let r′ ∈ Q, σ ∈ {R, N}, and w ∈ Γ^{∗} be such that

δ(r, a, A) = (r′, σ, w).  (3.1)

This transition means that if • the pushdown automaton is in state r,

• the tape head reads the symbol a, and

• the top symbol on the stack is A, then

• the pushdown automaton switches to state r′,

• the tape head moves according to σ: if σ = R, then it moves one cell to the right; if σ = N, then it does not move, and

• the top symbol A on the stack is replaced by the string w. We will write the computation step (3.1) in the form of the instruction

raA → r′σw.

We now specify the computation of the pushdown automaton M = (Σ, Γ, Q, δ, q).

Start configuration: Initially, the pushdown automaton is in the start state q, the tape head is on the leftmost symbol of the input string a_{1}a_{2} … a_{n}, and the stack contains only the special symbol $.

Computation and termination: Starting in the start configuration, the pushdown automaton performs a sequence of computation steps as described above. It terminates at the moment when the stack becomes empty. (Hence, if the stack never gets empty, the pushdown automaton does not terminate.)

Acceptance: The pushdown automaton accepts the input string a_{1}a_{2} … a_{n} ∈ Σ^{∗}, if

1. the automaton terminates on this input, and

2. at the time of termination (i.e., at the moment when the stack gets empty), the tape head is on the cell immediately to the right of the symbol a_{n} (this cell must contain the blank symbol □).

In all other cases, the pushdown automaton rejects the input string.

We denote by L(M), the language accepted by the pushdown automaton M; L(M) is the set of all strings in Σ^{∗} that are accepted by M.

The pushdown automaton described above is deterministic. For a nondeterministic pushdown automata, the next computation step may not be uniquely defined, but the automaton can make a choice out of a finite number of possibilities. In this case, the transition function δ is a function

δ : Q × (Σ ∪ {□}) × Γ → P_{f}(Q × {N, R} × Γ^{∗}),

where P_{f}(K) is the set of all finite subsets of the set K.

We say that a nondeterministic pushdown automaton M accepts an input string, if there exists an accepting computation, in the sense as described for deterministic pushdown automata. We say that M rejects an input string, if every computation on this string is rejecting. As before, we denote by L(M), the set of all strings in Σ^{∗} that are accepted by M.

3.6 Examples of pushdown automata

3.6.1 Properly nested parentheses

We will show how to construct a deterministic pushdown automaton, that accepts the set of all strings of properly nested parentheses. Observe that a string w in {(,)}^{∗} is properly nested if and only if

• in every prefix of w, the number of “(” is greater than or equal to the number of “)”, and

• in the complete string w, the number of “(” is equal to the number of “)”.

We will use the tape symbol a for “(”, and the tape symbol b for “)”.

The idea is as follows. Recall that initially, the stack contains only the special symbol $. The only purpose of this symbol is to enable us to check whether or not the stack is empty. The pushdown automaton reads the input string from left to right. For every a it reads, it pushes the symbol S onto the stack, and for every b it reads, it pops the top symbol from the stack. In this way, the number of symbols S on the stack will always be equal to the number of as that have been read minus the number of bs that have been read. The input string is properly nested if and only if this difference is always non-negative, and it is zero once the entire input string has been read. Hence, the input string is accepted if and only if during this process, the stack is never empty, and at the end, the stack contains only the special symbol $ (which will then be popped in the final step).

Based on this discussion, we obtain the deterministic pushdown automaton M = (Σ, Γ, Q, δ, q), where Σ = {a, b}, Γ = {$, S}, Q = {q}, and the transition function δ is specified by the following instructions:

qa$ → qR$S because of the a, S is pushed onto the stack
qaS → qRSS because of the a, S is pushed onto the stack
qbS → qRϵ  because of the b, the top element is popped

from the stack

qb$ → qNϵ  the number of bs read is larger than the number of as

read; the stack is made empty (hence, the computation terminates), and the input string is rejected

q□$ → qNϵ the input string has been read completely; the stack is

made empty, and the input string is accepted

q□S → qNS the input string has been read completely, it contains

more as than bs; no changes are made (so the

automaton does not terminate), and the input string is rejected

3.6.2 Strings of the form 0^{n}1^{n}

We construct a deterministic pushdown automata that accepts the language {0^{n}1^{n} : n ≥ 0}.

The automaton uses two states q_{0} and q_{1}, where q_{0} is the start state. Initially, the automaton is in state q_{0}.

• For each 0 that it reads, the automaton pushes one symbol S onto the stack and stays in state q_{0}.

• When the first 1 is read, the automaton switches to state q_{1}. From that moment,

– for each 1 that is read, the automaton pops the top symbol from the stack, and stays in state q_{1};

– if a 0 is read, the automaton does not make any change and, therefore, does not terminate.

Based on this discussion, we obtain the deterministic pushdown automaton M = (Σ, Γ, Q, δ, q_{0}), where Σ = {0, 1}, Γ = {$, S}, Q = {q_{0}, q_{1}}, q_{0} is the start state, and the transition function δ is specified by the following instructions:

q_{0}0$ → q_{0}R$S push S onto the stack
q_{0}0S → q_{0}RSS push S onto the stack
q_{0}1$ → q_{0}N$ first symbol in the input is 1; loop forever
q_{0}1S → q_{1}Rϵ first 1 is encountered
q_{0}□$ → q_{0}Nϵ input string is empty; accept it
q_{0}□S → q_{0}NS input consists only of 0s; loop forever
q_{1}0$ → q_{1}N$ 0 to the right of 1; loop forever
q_{1}0S → q_{1}NS 0 to the right of 1; loop forever
q_{1}1$ → q_{1}N$ too many 1s; loop forever
q_{1}1S → q_{1}Rϵ pop top symbol from the stack
q_{1}□$ → q_{1}Nϵ accept
q_{1}□S → q_{1}NS too many 0s; loop forever

3.6.3 Strings with b in the middle

We will construct a nondeterministic pushdown automaton that accepts the set L of all strings in {a, b}^{∗} having an odd length and whose middle symbol is b, i.e.,

L = {vbw : v ∈ {a, b}^{∗}, w ∈ {a, b}^{∗}, |v| = |w|}.

The idea is as follows. The automaton uses two states q and q′, where q is the start state. These states have the following meaning:

• If the automaton is in state q, then it has not reached the middle symbol of the input string.

• If the automaton is in state q′, then it has already read the middle symbol.

Observe that since the automaton can make only one single pass over the input string, it has to “guess” (i.e., use nondeterminism) when it reaches the middle of the string.

• If the automaton is in state q, then when reading the current input symbol,

– it pushes one symbol S onto the stack and stays in state q, or

– in case the current input symbol is b, it “guesses” that it has reached the middle of the input string, by switching to state q′.

• If the automaton is in state q′, then, when reading the current input symbol, it pops the top symbol S from the stack and stays in state q′.

The input string is accepted if and only if the stack is empty for the first time, after the entire input string has been read.

3.7. EQUIVALENCE OF PUSHDOWN AUTOMATA AND CONTEXT-FREE GRAMMARS101

We obtain the nondeterministic pushdown automaton M = (Σ, Γ, Q, δ, q), where Σ = {a, b}, Γ = {$, S}, Q = {q, q′}, q is the start state, and the transition function δ is specified by the following instructions: qa$ → qR$S qaS → qRSS

qb$ → q′R$  reached the middle
qb$ → qR$S not reached the middle
qbS → q′RS reached the middle
qbS → qRSS not reached the middle
q□$ → qN$ loop forever
q□S → qNS loop forever
q′a$ → q′Nϵ stack is empty; terminate, but reject, because

the input string has not been read completely

q′aS → q′Rϵ

q′b$ → q′Nϵ stack is empty; terminate, but reject, because

the input string has not been read completely

q′bS → q′Rϵ

q′□$ → q′Nϵ accept
q′□S → q′NS loop forever

Remark 3.6.1 It can be shown that there is no deterministic pushdown automaton that recognizes the language L. The reason is that a deterministic pushdown automaton cannot determine when it reaches the middle of the input string. Hence, unlike as for finite automata, nondeterministic pushdown automata are more powerful than their deterministic counterparts.

3.7 Equivalence of pushdown automata and context-free grammars

In this section, we will show that nondeterministic pushdown automata and context-free grammars are equivalent in power:

Theorem 3.7.1 Let Σ be an alphabet, and let A ⊂ Σ^{∗} be a language. Then A is context-free if and only if there exists a nondeterministic pushdown automaton that accepts A.

In fact, we will prove only one direction of this theorem. That is, we will show how to convert an arbitrary context-free grammar to a nondeterministic pushdown automaton.

Let G = (V, Σ, R, $) be a context-free grammar, where V is the set of variables, Σ is the set of terminals, R is the set of rules, and $ is the start variable. By Theorem 3.4.2, we may assume that G is in Chomsky normal form. Hence, every rule in R has one of the following three forms:

1. A → BC, where A, B, and C are variables, B ≠ S, and C ≠ S.

2. A → a, where A is a variable and a is a terminal,

3. $ → ϵ, where $ is the start variable.

We will construct a nondeterministic pushdown automaton M that accepts the language L(G) of this grammar G. Observe that M must have the following property: For every string w = a_{1}a_{2} … a_{n} ∈ Σ^{∗},

w ∈ L(G) if and only if M accepts w.

This can be reformulated as follows:

$ ⇒^{∗} a_{1}a_{2} … a_{n}

if and only if there exists a computation of M that starts in the initial configuration

[figure: a_{1}; ⋯; a_{i}; ⋯; a_{n}; □; $]

and ends in the configuration

[figure: a_{1}; ⋯; a_{i}; ⋯; a_{n}; □; ∅]

where ∅ indicates that the stack is empty.

Assume that $ ⇒^{∗} a_{1}a_{2} … a_{n}. Then there exists a derivation (using the rules of R) of the string a_{1}a_{2} … a_{n} from the start variable $. We may assume that in each step in this derivation, a rule is applied to the leftmost variable in the current string. Hence, at any moment during the derivation, the current string has the form

a_{1}a_{2} … a_{i−1}A_{k}A_{k−1} … A_{1},  (3.2)

for some integers i and k with 1 ≤ i ≤ n + 1 and k ≥ 0. (In particular, at the start of the derivation, we have i = 1 and k = 1, and the current string is A_{k} = $. At the end of the derivation, we have i = n + 1 and k = 0, and the current string is a_{1}a_{2} … a_{n}.)

We will define the pushdown automaton M in such a way that the current string (3.2) corresponds to the configuration

[figure: a_{1}; ⋯; a_{i}; ⋯; a_{n}; □; A_{1}; ⋮; A_{k}]

Based on this discussion, we obtain the nondeterministic pushdown automaton M = (Σ, V, {q}, δ, q), where

• the tape alphabet is the set Σ of terminals of G,

• the stack alphabet is the set V of variables of G,

• the set of states consists of one state q, which is the the start state, and

• the transition function δ is obtained from the rules in R, in the following way:

– For each rule in R that is of the form A → BC, with A, B, C ∈ V, the pushdown automaton M has the instructions

qaA → qNCB, for all a ∈ Σ.

– For each rule in R that is of the form A → a, with A ∈ V and a ∈ Σ, the pushdown automaton M has the instruction

qaA → qRϵ.

– If R contains the rule $ → ϵ, then the pushdown automaton M

has the instruction

q□$ → qNϵ.

This concludes the definition of M. It remains to prove that L(M) = L(G), i.e., the language of the nondeterministic pushdown automaton M is equal to the language of the context-free grammar G. Hence, we have to show that for every string w ∈ Σ^{∗},

w ∈ L(G) if and only if w ∈ L(M),

which can be rewritten as

$ ⇒^{∗} w if and only if M accepts w.

Claim 3.7.2 Let a_{1}a_{2} … a_{n} be a string in Σ^{∗}, let A_{1}, A_{2}, …, A_{k} be variables in V, and let i and k be integers with 1 ≤ i ≤ n + 1 and k ≥ 0. Then the following holds:

$ ⇒^{∗} a_{1}a_{2} … a_{i−1}A_{k}A_{k−1} … A_{1}

if and only if there exists a computation of M from the initial configuration

[figure: a_{1}; ⋯; a_{i}; ⋯; a_{n}; □; $]

to the configuration

[figure: a_{1}; ⋯; a_{i}; ⋯; a_{n}; □; A_{1}; ⋮; A_{k}]

Proof. The claim can be proved by induction. Let

w = a_{1}a_{2} … a_{i−1}A_{k}A_{k−1} … A_{1}.

Assume that k ≥ 1, and assume that the claim is true for the string w. Then you prove that the claim is still true after applying a rule in R to the leftmost variable A_{k} in w. Since the grammar is in Chomsky normal form, the rule to be applied is either of the form A_{k} → BC or of the form A_{k} → a_{i}. In both cases, the property mentioned in the claim is maintained.

We now use Claim 3.7.2 to prove that L(M) = L(G). Let w = a_{1}a_{2} … a_{n} be an arbitrary string in Σ^{∗}. By applying Claim 3.7.2, with i = n + 1 and k = 0, we see that w ∈ L(G), i.e.,

$ ⇒^{∗} a_{1}a_{2} … a_{n},

if and only if there exists a computation of M from the initial configuration

[figure: a_{1}; ⋯; a_{i}; ⋯; a_{n}; □; $]

to the configuration

[figure: a_{1}; ⋯; a_{i}; ⋯; a_{n}; □; ∅]

But this means that w ∈ L(G) if and only if the automaton M accepts the string w.

This concludes the proof of the fact that every context-free grammar can be converted to a nondeterministic pushdown automaton. As mentioned already, we will not give the conversion in the other direction. We finish this section with the following observation:

Theorem 3.7.3 Let Σ be an alphabet, and let A ⊂ Σ^{∗} be a context-free language. Then there exists a nondeterministic pushdown automaton that accepts A and that has only one state.

Proof. Since A is context-free, there exists a context-free grammar G_{0} such that L(G_{0}) = A. By Theorem 3.4.2, there exists a context-free grammar G that is in Chomsky normal form and for which L(G) = L(G_{0}). The construction given above converts G to a nondeterministic pushdown automaton M that has only one state and for which L(M) = L(G).

3.8 The pumping lemma for context-free languages

In Section 2.9, we proved the pumping lemma for regular languages, and used it to prove that certain languages are not regular. In this section, we generalize the pumping lemma to context-free languages. The idea is to consider the parse tree (see Section 3.1) that describes the derivation of a sufficiently long string in the context-free language A. Since the number of variables in the corresponding context-free grammar G is finite, there is at least one variable, say A_{j}, that occurs more than once on the longest rootto-leaf path in the parse tree. The subtree which is sandwiched between two occurrences of A_{j} on this path can be copied any number of times. This will result in a legal parse tree and, hence, in a “pumped” string that is in the language A.

Theorem 3.8.1 (Pumping Lemma for Context-Free Languages) Let A be a context-free language. Then there exists an integer p ≥ 1, called the pumping length, such that the following holds: Every string s in A, with |s| ≥ p, can be written as s = uvxyz, such that

1. |vy| ≥ 1 (i.e., v and y are not both empty),

2. |vxy| ≤ p, and

3. uv^{i}xy^{i}z ∈ A, for all i ≥ 0.

3.8.1 Proof of the pumping lemma

The proof of the pumping lemma will use the following result about parse trees:

Lemma 3.8.2 Let G be a context-free grammar in Chomsky normal form, let s be a non-empty string in L(G), and let T be a parse tree for s. Let ℓ be the number of edges on a longest root-to-leaf path in T. Then

|s| ≤ 2^{ℓ−1}.

Proof. The claim can be proved by induction on ℓ. By looking at some small values of ℓ, you should be able to see why the claim holds.

3.8. THE PUMPING LEMMA FOR CONTEXT-FREE LANGUAGES107

Now we can start with the proof of the pumping lemma. Let A be a context-free language. Then, by Theorem 3.4.2, there exists a context-free grammar in Chomsky normal form, G = (V, Σ, R, S), such that A = L(G).

Define r to be the number of variables of G, and p := 2^{r}. We will prove that the value of p can be used as the pumping length. Consider an arbitrary string s in A, such that |s| ≥ p, and let T be a parse tree for s. Let ℓ be the number of edges on a longest root-to-leaf path in T. Then, by Lemma 3.8.2, we have

|s| ≤ 2^{ℓ−1}.

On the other hand, we have

|s| ≥ p = 2^{r}.

By combining these inequalities, we see that 2^{r} ≤ 2^{ℓ−1}, which can be rewritten as

ℓ ≥ r + 1.

Consider the nodes on a longest root-to-leaf path in T. Since this path consists of ℓ edges, it consists of ℓ + 1 nodes. The first ℓ of these nodes store variables, which we denote by A_{0}, A_{1}, …, A_{ℓ−1} (where A_{0} = S), and the last node (which is a leaf) stores a terminal, which we denote by a.

Since ℓ − 1 − r ≥ 0, the sequence

A_{ℓ−1−r}, A_{ℓ−r}, …, A_{ℓ−1}

of variables is well-defined. Observe that this sequence consists of r + 1 variables. Since the number of variables in the grammar G is equal to r, the pigeon hole principle implies that there is a variable that occurs at least twice in this sequence. In other words, there are indices j and k, such that ℓ − 1 − r ≤ j < k ≤ ℓ − 1 and A_{j} = A_{k}. Refer to the figure below for an illustration.

S A_{j} A_{k} u  v  x  y  z s A_{0} = SA_{1} A_{ℓ−1−r} A_{ℓ−r} A_{ℓ−2} A_{ℓ−1} a  r + 1 variables

Recall that T is a parse tree for the string s. Therefore, the terminals stored at the leaves of T, in the order from left to right, form s. As indicated in the figure above, the nodes storing the variables A_{j} and A_{k} partition s into five substrings u, v, x, y, and z, such that s = uvxyz.

It remains to prove that the three properties stated in the pumping lemma hold. We start with the third property, i.e., we prove that

uv^{i}xy^{i}z ∈ A, for all i ≥ 0.

In the grammar G, we have

S ⇒^{∗} uA_{j}z.  (3.3)

Since A_{j} ⇒^{∗} vA_{k}y and A_{k} = A_{j}, we have

A_{j} ⇒^{∗} vA_{j}y.  (3.4)

Finally, since A_{k} ⇒^{∗} x and A_{k} = A_{j}, we have

A_{j} ⇒^{∗} x.  (3.5)

From (3.3) and (3.5), it follows that

S ⇒^{∗} uA_{j}z ⇒^{∗} uxz,

which implies that the string uxz is in the language A. Similarly, it follows from (3.3), (3.4), and (3.5) that

S ⇒^{∗} uA_{j}z ⇒^{∗} uvA_{j}yz ⇒^{∗} uvvA_{j}yyz ⇒^{∗} uvvxyyz.

Hence, the string uv^{2}xy^{2}z is in the language A. In general, for each i ≥ 0, the string uv^{i}xy^{i}z is in the language A, because

S ⇒^{∗} uA_{j}z ⇒^{∗} uv^{i}A_{j}y^{i}z ⇒^{∗} uv^{i}xy^{i}z.

This proves that the third property in the pumping lemma holds.

Next we show that the second property holds. That is, we prove that |vxy| ≤ p. Consider the subtree rooted at the node storing the variable A_{j}. The path from the node storing A_{j} to the leaf storing the terminal a is a longest path in this subtree. (Convince yourself that this is true). Moreover, this path consists of ℓ − j edges. Since A_{j} ⇒^{∗} vxy, this subtree is a parse tree for the string vxy (where A_{j} is used as the start variable). Therefore, by Lemma 3.8.2, we can conclude that |vxy| ≤ 2^{ℓ−j−1}. We know that ℓ − 1 − r ≤ j, which is equivalent to ℓ − j − 1 ≤ r. It follows that

|vxy| ≤ 2^{ℓ−j−1} ≤ 2^{r} = p.

Finally, we show that the first property in the pumping lemma holds. That is, we prove that |vy| ≥ 1. Recall that

A_{j} ⇒^{∗} vA_{k}y.

Let the first rule used in this derivation be A_{j} → BC. (Since the variables A_{j} and A_{k}, even though they are equal, are stored at different nodes of the parse tree, and since the grammar G is in Chomsky normal form, this first rule exists.) Then

A_{j} ⇒ BC ⇒^{∗} vA_{k}y.

Observe that the string BC has length two. Moreover, by applying rules of a grammar in Chomsky normal form, strings cannot become shorter. (Here, we use the fact that the start variable does not occur on the right-hand side of any rule.) Therefore, we have |vA_{k}y| ≥ 2. But this implies that |vy| ≥ 1. This completes the proof of the pumping lemma.

3.8.2 Applications of the pumping lemma

First example

Consider the language

A = {a^{n}b^{n}c^{n} : n ≥ 0}.

We will prove by contradiction that A is not a context-free language.

Assume that A is a context-free language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Consider the string s = a^{p}b^{p}c^{p}. Observe that s ∈ A and |s| = 3p ≥ p. Hence, by the pumping lemma, s can be written as s = uvxyz, where |vy| ≥ 1, |vxy| ≤ p, and uv^{i}xy^{i}z ∈ A for all i ≥ 0.

Observe that, since |vxy| ≤ p, the string vxy cannot contain all three different symbols a, b, and c. In fact, there is a symbol α ∈ {a, c} that does not occur in v and that does not occur in y. Hence, in the string uxz, the symbol α occurs exactly p times. Since |vy| ≥ 1, we have |uxz| < |uvxyz| = |s| = 3p. Therefore, in the string uxz, the symbol α occurs exactly p times, whereas the total number of occurrences of the other two symbols is less than 2p. This implies that there is a symbol that occurs less than p times in the string uxz. This means that the string uv^{0}xy^{0}z = uxz is not contained in A. But, by the pumping lemma, this string is contained in A. This is a contradiction and, therefore, we have shown that the language A is not context-free.

Second example

Consider the languages

A = {ww^{R} : w ∈ {a, b}^{∗}},

where w^{R} is the string obtained by writing w backwards, and

B = {ww : w ∈ {a, b}^{∗}}.

Even though these languages look similar, we will show that A is context-free, whereas B is not context-free.

Consider the following context-free grammar, in which S is the start variable:

S → ϵ|aSa|bSb.

It is easy to see that the language of this grammar is exactly the language A. Therefore, A is context-free. Alternatively, we can show that A is contextfree, by constructing a (nondeterministic) pushdown automaton that accepts A. This automaton has two states q and q′, where q is the start state. If the automaton is in state q, then it did not yet read the leftmost half of the input string; it pushes all symbols read onto the stack. If the automaton is in state q′, then it is reading the rightmost half of the input string; for each symbol read, it checks whether it is equal to the symbol on top of the stack and, if so, pops the top symbol from the stack. The pushdown automaton uses nondeterminism to “guess” when to switch from state q to state q′ (i.e., when it has completed reading the leftmost half of the input string).

At this point, you should convince yourself that the two approaches above, which showed that A is context-free, do not work for B. The reason why they do not work is that the language B is not context-free, as we will prove now.

Assume that B is a context-free language. Let p ≥ 1 be the pumping length, as given by the pumping lemma. At this point, we must choose a string s in B, whose length is at least p, and that does not satisfy the three properties stated in the pumping lemma. Let us try the string s = a^{p}ba^{p}b. Then s ∈ B and |s| = 2p + 1 ≥ p. Hence, by the pumping lemma, s can be written as s = uvxyz, where (i) |vy| ≥ 1, (ii) |vxy| ≤ p, and (iii) uv^{i}xy^{i}z ∈ B for all i ≥ 0. It may happen that p ≥ 3, and u = a^{p−1}, v = a, x = b, y = a, and z = a^{p−1}b. If this is the case, then properties (i), (ii), and (iii) hold, and, thus, we do not get a contradiction. In other words, we have chosen the “wrong” string s. This string is “wrong”, because there is only one b between the as. Because of this, v can be in the leftmost block of as, and y can be in the rightmost block of as. Observe that if there were at least p many bs between the as, then this cannot happen, because |vxy| ≤ p.

Based on the discussion above, we choose s = a^{p}b^{p}a^{p}b^{p}. Observe that s ∈ B and |s| = 4p ≥ p. Hence, by the pumping lemma, s can be written as s = uvxyz, where |vy| ≥ 1, |vxy| ≤ p, and uv^{i}xy^{i}z ∈ B for all i ≥ 0. Based on the location of vxy in the string s, we distinguish three cases:

Case 1: The substring vxy overlaps both the leftmost half and the rightmost half of s.

Since |vxy| ≤ p, the substring vxy is contained in the “middle” part of s, i.e., vxy is contained in the block b^{p}a^{p}. Consider the string uv^{0}xy^{0}z = uxz. Since |vy| ≥ 1, we know that at least one of v and y is non-empty.

• If v ≠ ϵ, then v contains at least one b from the leftmost block of bs in s, whereas y does not contain any b from the rightmost block of bs in s. Therefore, in the string uxz, the leftmost block of bs contains fewer bs than the rightmost block of bs. Hence, the string uxz is not contained in B.

• If y ≠ ϵ, then y contains at least one a from the rightmost block of as in s, whereas v does not contain any a from the leftmost block of as in s. Therefore, in the string uxz, the leftmost block of as contains more as than the rightmost block of as. Hence, the string uxz is not contained in B.

In both cases, we conclude that the string uxz is not an element of the language B. But, by the pumping lemma, this string is contained in B.

Case 2: The substring vxy is in the leftmost half of s.

In this case, none of the strings uxz, uv^{2}xy^{2}z, uv^{3}xy^{3}z, uv^{4}xy^{4}z, etc., is contained in B. But, by the pumping lemma, each of these strings is contained in B.

Case 3: The substring vxy is in the rightmost half of s.

This case is symmetric to Case 2: None of the strings uxz, uv^{2}xy^{2}z, uv^{3}xy^{3}z, uv^{4}xy^{4}z, etc., is contained in B. But, by the pumping lemma, each of these strings is contained in B.

To summarize, in each of the three cases, we have obtained a contradiction. Therefore, the language B is not context-free.

Third example

We have seen in Section 3.2.4 that the language

{a^{m}b^{n}c^{m+n} : m ≥ 0, n ≥ 0}

is context-free. Using the pumping lemma for regular languages, it is easy to prove that this language is not regular. In other words, context-free grammars can verify addition, whereas finite automata are not powerful enough for this. We now consider the problem of verifying multiplication: Let A be the language defined as

A = {a^{m}b^{n}c^{mn} : m ≥ 0, n ≥ 0}.

We will prove by contradiction that A is not a context-free language.

Assume that A is context-free. Let p ≥ 1 be the pumping length, as given by the pumping lemma. Consider the string s = a^{p}b^{p}c^{p^{2}}. Then, s ∈ A and |s| = 2p + p^{2} ≥ p. Hence, by the pumping lemma, s can be written as s = uvxyz, where |vy| ≥ 1, |vxy| ≤ p, and uv^{i}xy^{i}z ∈ A for all i ≥ 0.

There are three possible cases, depending on the location of vxy in the string s.

Case 1: The substring vxy is completely contained in the block of cs in s.

Consider the string uv^{2}xy^{2}z. Since |vy| ≥ 1, the string uv^{2}xy^{2}z consists of p many as, p many bs, but more than p^{2} many cs. Therefore, this string is not contained in A. But, by the pumping lemma, it is contained in A.

Case 2: The substring vxy does not contain any c.

Consider again the string uv^{2}xy^{2}z. This string consists of p^{2} many cs. Since |vy| ≥ 1, in the string uv^{2}xy^{2}z, the number of as multiplied by the number of bs is larger than p^{2}. Therefore, uv^{2}xy^{2}z is not contained in A. But, by the pumping lemma, this string is contained in A.

Case 3: The substring vxy contains at least one b and at least one c.

In this case, we can write vy = b^{j}c^{k}, where j ≥ 0, k ≥ 0, and j + k ≥ 1. Consider the string uxz. We can write this string as uxz = a^{p}b^{p−j}c^{p^{2}−k}. Since, by the pumping lemma, this string is contained in A, we have p(p−j) = p^{2} − k, which implies that jp = k.

If j = 0, then k = 0, which contradicts the fact that j +k ≥ 1. Therefore, j ≥ 1. It follows that k = jp ≥ p and

|vxy| ≥ |vy| = j + k ≥ 1 + p.

But, by the pumping lemma, we have |vxy| ≤ p.

Observe that, since |vxy| ≤ p, the above three cases cover all possibilities for the location of vxy in the string s. In each of the three cases, we have obtained a contradiction. Therefore, the language A is not context-free.

3.9 Exercises

3.1 Construct context-free grammars that generate the following languages. In all cases, Σ = {0, 1}.

• {w : w contains at least three 1s} 114 CHAPTER 3. CONTEXT-FREE LANGUAGES

• {w : the length of w is odd and its middle symbol is 0}

• {w : w is a palindrome}. A palindrome is a string w having the property that w = w^{R}, i.e., reading w from left to right gives the same result as reading w from right to left.

• {w : w starts and ends with the same symbol} 3.2 Let G = (V, Σ, R, S) be the context-free grammar, where V = {A, B, S}, Σ = {0, 1}, S is the start variable, and R consists of the rules

\begin{aligned}S & → & 0S|1A|ϵ \\ A & → & 0B|1S \\ B & → & 0A|1B \end{aligned}

Define the following language L:

L := {w ∈ {0, 1}^{∗} : w is the binary representation of a non-negative integer

that is divisible by three } ∪ {ϵ}

Prove that L = L(G). (Hint: The variables S, A, and B are used to remember the remainder after division by three.) 3.3 Let A and B be context-free languages over the same alphabet Σ.

• Prove that the union A ∪ B of A and B is also context-free.

• Prove that the concatenation AB of A and B is also context-free.

• Prove that the star A^{∗} of A is also context-free. 3.4 Define the following two languages A and B:

A = {a^{m}b^{n}c^{n} : m ≥ 0, n ≥ 0}

and

B = {a^{m}b^{m}c^{n} : m ≥ 0, n ≥ 0}.

• Prove that both A and B are context-free, by constructing two grammars, one that generates A and one that generates B.

• We have seen in Section 3.8.2 that the language

{a^{n}b^{n}c^{n} : n ≥ 0}

is not context-free. Explain why this implies that the intersection of two context-free languages is not necessarily context-free.

• Use De Morgan’s Law to conclude that the complement of a contextfree language is not necessarily context-free.

3.5 Let L be a language consisting of finitely many strings. Show that L is regular and, therefore, context-free. Let k be the maximum length of any string in L.

• Prove that every context-free grammar in Chomsky Normal Form that generates L has more than log k variables. (The logarithm is in base 2.)

• Prove that there is a context-free grammar that generates L and that has only one variable.

3.6 Construct (deterministic or nondeterministic) pushdown automata that accept the following languages.

1. {w ∈ {0, 1}^{∗} : w contains more 1s than 0s}.

2. {w ∈ {0, 1}^{∗} : w is a palindrome}.

3. {w ∈ {a, b}^{∗} : number of as is at least equal to number of bs and is at most twice the number of bs.}

3.7 Prove that the following languages are not context-free.

• {a^{n}b^{n}a^{n}b^{n} : n ≥ 0}.

• {w#x| w is a substring of x, and w, x ∈ {a, b}^{∗}}. For example, aba#abbababbb is a string in the language, whereas aba#baabbaabb is not a string in the language. The alphabet is {a, b, #}.

• {1^{n} : n is a prime number}.

• {a^{3k}b^{2k}c^{k}|k = 0, 1, 2, 3, ⋯ }.

3.8 Prove that the intersection of a context free language L with a regular language M is a context free language. The languages L and M are defined over the same alphabet set Σ.

3.9 Suppose that for a pushdown automaton M the size of the stack never grows more than a fixed constant k. Show that the language accepted by M is a regular language.

3.10 Convert the following context free grammars to the Chomsky Normal Form.

• S → SS, S → (S), S → ϵ; Σ = {(,)}.

• S → aSb, S → bSa, S → ϵ; Σ = {a, b}.

• S → AB, A → aAa, B → bBb, A → ϵ, B → ϵ; Σ = {a, b}.

3.11 We know that the language L = {x ∈ {0, 1}^{∗}|x = ww for any w ∈ {0, 1}^{∗}} is not a context free language as shown in the Second Example in Section 3.8.2. What about the complement of L? Show that the complement of L is a context free language. (Hint: You can think of the complement as the union of the languages consisting of odd number of strings or languages of the form u0v1w or u1v0w, where |v| = |u| + |w|.

3.12 If a context free grammar G is given in the Chomsky Normal Form, then show that for any string w ∈ L(G), exactly 2|w| − 1 steps are required for any derivation of w.

3.13 In one of the exercises above we showed that context free languages are closed under the operation of union, concatenation and star. Using this show how regular expressions can be converted to an equivalent context free grammars.

3.14 Construct a CFG to generate all regular expressions over the alphabet Σ = {a, b}.

Turing Machines and the Church-Turing Thesis

In the previous chapters, we have seen several computational devices that can be used to accept or generate regular and context-free languages. Even though these two classes of languages are fairly large, we have seen in Section 3.8.2 that these devices are not powerful enough to accept simple languages such as A = {a^{m}b^{n}c^{mn} : m ≥ 0, n ≥ 0}. In this chapter, we introduce the Turing machine, which is a simple model of a real computer. Turing machines can be used to accept all context-free languages, but also languages such as A. We will argue that every problem that can be solved on a real computer can also be solved by a Turing machine (this statement is known as the Church-Turing Thesis). In Chapter 5, we will consider the limitations of Turing machines and, hence, of real computers.

4.1 Definition of a Turing machine

We start with an informal description of a Turing machine. Such a machine consists of the following, see also Figure 4.1.

1. There are k tapes, for some fixed k ≥ 1. Each tape is divided into cells, and is infinite both to the left and to the right. Each cell stores a symbol belonging to a finite set Γ, which is called the tape alphabet. The tape alphabet contains the blank symbol □. If a cell contains □, then this means that the cell is actually empty.

118CHAPTER 4. TURING MACHINES AND THE CHURCH-TURING THESIS

finite control

… □ □ □ a a b a b b a b a b □ □ □ …

… □ □ □ b a a b □ a b □ □ □ …

Figure 4.1: A Turing machine with k = 2 tapes.

2. Each tape has a tape head which can move along the tape, one cell per move. It can also read the cell it currently scans and replace the symbol in this cell by another symbol.

3. There is a finite control, which can be in any one of a finite number of states. The finite set of states is denoted by Q. The set Q contains three special states: a start state, an accept state, and a reject state.

The Turing machine performs a sequence of computation steps. In one such step, it does the following:

1. Immediately before the computation step, the Turing machine is in a state r of Q, and each of the k tape heads is on a certain cell.

2. Depending on the current state r and on the k symbols that are read by the tape heads,

(a) the Turing machine switches to a state r′ of Q (which may be equal to r),

(b) each tape head writes a symbol of Γ in the cell it is currently scanning (this symbol may be equal to the symbol currently stored in the cell), and 4.1. DEFINITION OF A TURING MACHINE 119

(c) each tape head either moves one cell to the left, moves one cell to

the right, or stays at the current cell.

We now give a formal definition of a deterministic Turing machine. Definition 4.1.1 A deterministic Turing machine is a 7-tuple M = (Σ, Γ, Q, δ, q, q_{accept}, q_{reject}), where

1. Σ is a finite set, called the input alphabet; the blank symbol □ is not contained in Σ,

2. Γ is a finite set, called the tape alphabet; this alphabet contains the blank symbol □, and Σ ⊆ Γ,

3. Q is a finite set, whose elements are called states,

4. q is an element of Q; it is called the start state,

5. q_{accept} is an element of Q; it is called the accept state,

6. q_{reject} is an element of Q; it is called the reject state,

7. δ is called the transition function, which is a function δ : Q × Γ^{k} → Q × Γ^{k} × {L, R, N}^{k}.

The transition function δ is basically the “program” of the Turing machine. This function tells us what the machine can do in “one computation step”: Let r ∈ Q, and let a_{1}, a_{2}, …, a_{k} ∈ Γ. Furthermore, let r′ ∈ Q, a′_{1}, a′_{2}, …, a′_{k} ∈ Γ, and σ_{1}, σ_{2}, …, σ_{k} ∈ {L, R, N} be such that

δ(r, a_{1}, a_{2}, …, a_{k}) = (r′, a′_{1}, a′_{2}, …, a′_{k}, σ_{1}, σ_{2}, …, σ_{k}).  (4.1)

This transition means that if • the Turing machine is in state r, and

• the head of the i-th tape reads the symbol a_{i}, 1 ≤ i ≤ k, then

• the Turing machine switches to state r′, • the head of the i-th tape replaces the scanned symbol a_{i} by the symbol

a′_{i}, 1 ≤ i ≤ k, and

• the head of the i-th tape moves according to σ_{i}, 1 ≤ i ≤ k: if σ_{i} = L, then the tape head moves one cell to the left; if σ_{i} = R, then it moves one cell to the right; if σ_{i} = N, then the tape head does not move.

Usually, we will write the computation step (4.1) in the form of the instruction

ra_{1}a_{2} … a_{k} → r′a′_{1}a′_{2} … a′_{k}σ_{1}σ_{2} … σ_{k}.

We now specify the computation of the Turing machine M = (Σ, Γ, Q, δ, q, q_{accept}, q_{reject}).

Start configuration: The input is a string over the input alphabet Σ. Initially, this input string is stored on the first tape, and the head of this tape is on the leftmost symbol of the input string. Initially, all other k − 1 tapes are empty, i.e., contain only blanks, and the Turing machine is in the start state q.

Computation and termination: Starting in the start configuration, the Turing machine performs a sequence of computation steps as described above. The computation terminates at the moment when the Turing machine enters the accept state q_{accept} or the reject state q_{reject}. (Hence, if the Turing machine never enters the states q_{accept} and q_{reject}, the computation does not terminate.)

Acceptance: The Turing machine M accepts the input string w ∈ Σ^{∗}, if the computation on this input terminates in the state q_{accept}. If the computation on this input terminates in the state q_{reject}, then M rejects the input string w.

We denote by L(M), the language accepted by the Turing machine M; L(M) is the set of all strings in Σ^{∗} that are accepted by M.

4.2 Examples of Turing machines

4.2.1 Accepting palindromes using one tape

We will show how to construct a Turing machine with one tape, that decides whether or not any input string w ∈ {a, b}^{∗} is a palindrome. Recall that the string w is called a palindrome, if reading w from left to right gives the same result as reading w from right to left. Examples of palindromes are abba, baabbbbaab and the empty string^{1}.

Start of the computation: The tape contains the input string w, the tape head is on the leftmost symbol of w, and the Turing machine is in the start state q_{0}.

Idea: The tape head reads the leftmost symbol of w, deletes this symbol and “remembers” it by means of a state. Then the tape head moves to the rightmost symbol and tests whether it is equal to the (already deleted) leftmost symbol.

• If they are equal, then the rightmost symbol is deleted, the tape head moves to the new leftmost symbol, and the whole process is repeated.

• If they are not equal, the Turing machine enters the reject state, and the computation terminates.

The Turing machine enters the accept state as soon as the string currently stored on the tape is empty.

We will use the input alphabet Σ = {a, b} and the tape alphabet Γ = {a, b, □}. The set Q of states consists of the following eight states:

q_{0} :  start state; tape head is on the leftmost symbol
q_{a} :  leftmost symbol was a; tape head is moving to the right
q_{b} :  leftmost symbol was b; tape head is moving to the right
q′_{a} :  reached rightmost symbol; test whether it is equal to a, and delete it
q′_{b} :  reached rightmost symbol; test whether it is equal to b, and delete it
q_{2} :  test was positive; tape head is moving to the left
q_{accept} : accept state
q_{reject} : reject state

The transition function δ is specified by the following instructions:

\begin{matrix}q_{0}a → q_{a}□R & q_{a}a → q_{a}aR & q_{b}a → q_{b}aR \\ q_{0}b → q_{b}□R & q_{a}b → q_{a}bR & q_{b}b → q_{b}bR \\ q_{0}□ → q_{accept} & q_{a}□ → q′_{a}□L & q_{b}□ → q′_{b}□L \end{matrix}

1Another example is: Bob: “Did Anna peep?” Anna: “Did Bob?”. See the web page http://www.palindromes.org/

\begin{matrix}q′_{a}a → q_{2}□L & q′_{b}a → q_{reject} & q_{2}a → q_{2}aL \\ q′_{a}b → q_{reject} & q′_{b}b → q_{2}□L & q_{2}b → q_{2}bL \\ q′_{a}□ → q_{accept} & q′_{b}□ → q_{accept} & q_{2}□ → q_{0}□R \end{matrix}

You should go through the computation of this Turing machine for some sample inputs, for example abba, b, abb and the empty string (which is a palindrome).

4.2.2 Accepting palindromes using two tapes

We again consider the palindrome problem, but now we use a Turing machine with two tapes.

Start of the computation: The first tape contains the input string w, and the head of the first tape is on the leftmost symbol of w. The second tape is empty, and its tape head is at an arbitrary position. The Turing machine is in the start state q_{0}.

Idea: First, the input string w is copied to the second tape. Then the head of the first tape moves back to the leftmost symbol of w, while the head of the second tape stays at the rightmost symbol of w. Finally, the actual test starts: The head of the first tape moves to the right and, at the same time, the head of the second tape moves to the left. While moving, the Turing machine tests whether the two tape heads read the same symbol in each step.

The input alphabet is Σ = {a, b}, and the tape alphabet is Γ = {a, b, □}. The set Q of states consists of the following five states:

q_{0} :  start state; copy w to the second tape
q_{1} :  w has been copied; head of first tape moves to the left
q_{2} :  head of first tape moves to the right; head of second tape moves

to the left; until now, all tests were positive

q_{accept} : accept state
q_{reject} : reject state

4.2. EXAMPLES OF TURING MACHINES 123

The transition function δ is specified by the following instructions:

q_{0}a□ → q_{0}aaRR  q_{1}aa → q_{1}aaLN
q_{0}b□ → q_{0}bbRR  q_{1}ab → q_{1}abLN
q_{0}□□ → q_{1}□□LL  q_{1}ba → q_{1}baLN
q_{1}bb → q_{1}bbLN
q_{1}□a → q_{2}□aRN
q_{1}□b → q_{2}□bRN
q_{1}□□ → q_{accept}
q_{2}aa → q_{2}aaRL
q_{2}ab → q_{reject}
q_{2}ba → q_{reject}
q_{2}bb → q_{2}bbRL
q_{2}□□ → q_{accept}

Again, you should run this Turing machine for some sample inputs.

4.2.3 Accepting a^{n}b^{n}c^{n} using one tape

We will construct a Turing machine with one tape that accepts the language

{a^{n}b^{n}c^{n} : n ≥ 0}.

Recall that we have proved in Section 3.8.2 that this language is not contextfree. Start of the computation: The tape contains the input string w, and the tape head is on the leftmost symbol of w. The Turing machine is in the start state q_{0}. Idea: Repeat the following Stages 1 and 2, until the string is empty. Stage 1. Walk along the string from left to right, delete the leftmost a, delete the leftmost b, and delete the rightmost c. Stage 2. Shift the substring of bs and cs one position to the left.

The input alphabet is Σ = {a, b, c}, and the tape alphabet is Γ = {a, b, c, □}. For Stage 1, we use the following states:

q_{0} :  start state; tape head is on the leftmost symbol
q_{a} :  leftmost a has been deleted; have not read b
q_{b} :  leftmost b has been deleted; have not read c
q_{c} :  leftmost c has been read; tape head moves to the right
q′_{c} :  tape head is on the rightmost c
q_{1} :  rightmost c has been deleted; tape head is on the rightmost

symbol or □

q_{accept} : accept state
q_{reject} : reject state

The transitions for Stage 1 are specified by the following instructions:

\begin{matrix}q_{0}a → q_{a}□R & q_{a}a → q_{a}aR \\ q_{0}b → q_{reject} & q_{a}b → q_{b}□R \\ q_{0}c → q_{reject} & q_{a}c → q_{reject} \\ q_{0}□ → q_{accept} & q_{a}□ → q_{reject} \end{matrix}

q_{b}a → q_{reject}  q_{c}a → q_{reject}
q_{b}b → q_{b}bR  q_{c}b → q_{reject}
q_{b}c → q_{c}cR  q_{c}c → q_{c}cR
q_{b}□ → q_{reject}  q_{c}□ → q′_{c}□L
q′_{c}c → q_{1}□L

For Stage 2, we use the following states:

q_{1} : as above; tape head is on the rightmost symbol or on □
q^{c} : copy c one cell to the left
q^{b} : copy b one cell to the left
q_{2} : done with Stage 2; head moves to the left

Additionally, we use a state q′_{1} which has the following meaning: If the input string is of the form a^{i}bc, for some i ≥ 1, then after Stage 1, the tape contains the string a^{i−1}□□, the tape head is on the □ immediately to the right of the as, and the Turing machine is in state q_{1}. In this case, we move one cell to the left; if we then read □, then i = 1, and we accept; otherwise, we read a, and we reject.

The transitions for Stage 2 are specified by the following instructions:

q_{1}a → cannot happen  q′_{1}a → q_{reject}
q_{1}b → q_{reject}  q′_{1}b → cannot happen
q_{1}c → q^{c}□L  q′_{1}c → cannot happen
q_{1}□ → q′_{1}□L  q′_{1}□ → q_{accept}
q^{c}a → cannot happen  q^{b}a → cannot happen
q^{c}b → q^{b}cL  q^{b}b → q^{b}bL
q^{c}c → q^{c}cL  q^{b}c → cannot happen
q^{c}□ → q_{reject}  q^{b}□ → q_{2}bL
q_{2}a → q_{2}aL
q_{2}b → cannot happen
q_{2}c → cannot happen
q_{2}□ → q_{0}□R

4.2.4 Accepting a^{m}b^{n}c^{mn} using one tape

We will sketch how to construct a Turing machine with one tape that accepts the language

{a^{m}b^{n}c^{mn} : m ≥ 0, n ≥ 0}.

Recall that we have proved in Section 3.8.2 that this language is not contextfree.

Start of the computation: The tape contains the input string w, and the tape head is on the leftmost symbol of w. The Turing machine is in the start state.

Idea: Observe that a string a^{m}b^{n}c^{k} is in the language if and only if for every a, the string contains exactly n many cs. Based on this, the computation consists of the following stages:

Stage 1. Walk along the input string w from left to right, and check whether w is an element of a^{∗}b^{∗}c^{∗}. If this is not the case, then reject the input string. Otherwise, go to Stage 2.

Stage 2. Walk back to the leftmost symbol of w. Go to Stage 3.

Stage 3. In this stage, the Turing machine does the following:

• Replace the leftmost a by the blank symbol □.

• Walk to the leftmost b.

• Zigzag between the bs and cs; each time, replace the leftmost b by the symbol $, and replace the rightmost c by the blank symbol □. If, for some b, there is no c left, the Turing machine rejects the input string.

• Continue zigzagging until there are no bs left. Then go to Stage 4

Observe that in this third stage, the string a^{m}b^{n}c^{k} is transformed to the string a^{m−1}$^{n}c^{k−n}.

Stage 4. In this stage, the Turing machine does the following:

• Replace each $ by b.

• Walk to the leftmost a.

Hence, in this fourth stage, the string a^{m−1}$^{n}c^{k−n} is transformed to the string a^{m−1}b^{n}c^{k−n}.

Observe that the input string a^{m}b^{n}c^{k} is in the language if and only if the string a^{m−1}b^{n}c^{k−n} is in the language. Therefore, the Turing machine repeats Stages 3 and 4, until there are no as left. At that moment, it checks whether there are any cs left; if so, it rejects the input string; otherwise, it accepts the input string.

We hope you believe that this description of the algorithm can be turned into a formal description of a Turing machine.

4.3 Multi-tape Turing machines

In Section 4.2, we have seen two Turing machines that accept palindromes; the first Turing machine has one tape, whereas the second one has two tapes. You will have noticed that the two-tape Turing machine was easier to obtain than the one-tape Turing machine. This leads to the question whether multitape Turing machines are more powerful than their one-tape counterparts. The answer is “no”:

Theorem 4.3.1 Let k ≥ 1 be an integer. Any k-tape Turing machine can be converted to an equivalent one-tape Turing machine.

Proof. We will sketch the proof for the case when k = 2. Let M = (Σ, Γ, Q, δ, q, q_{accept}, q_{reject}) be a two-tape Turing machine. Our goal is to convert M to an equivalent one-tape Turing machine N. That is, N should have the property that for all strings w ∈ Σ^{∗},

• M accepts w if and only if N accepts w,

• M rejects w if and only if N rejects w,

• M does not terminate on input w if and only if N does not terminate on input w.

The tape alphabet of the one-tape Turing machine N is

Γ ∪ {ẋ : x ∈ Γ} ∪ {#}.

In words, we take the tape alphabet Γ of M, and add, for each x ∈ Γ, the symbol ẋ. Moreover, we add a special symbol #.

The Turing machine N will be defined in such a way that any configuration of the two-tape Turing machine M, for example

[figure: …; □; 1; 0; 0; 1; □; …; …; □; a; a; b; a; □; …]

corresponds to the following configuration of the one-tape Turing machine N:

… □ # 1 0 ˙0 1 # a ȧ b a # □ …^{}

The Turing machine N simulates one computation step of M, as follows:

• At the start of the simulation, the tape head of N is on the leftmost symbol #.

• N walks along the string to the right, and finds the first dotted symbol. (This symbol indicates the location of the head of the first tape of M.) Then, it updates this part of the tape, by making the change that M would make on its first tape. (This change is given by the transition function of M.)

• Next, N continues to walk to the right, and finds the second dotted symbol. (This symbol indicates the location of the head of the second tape of M.) Then, it updates this part of the tape, by making the change that M would make on its second tape.

• In the two steps above, it may be necessary to shift a part of the tape.

• Finally, N walks back to the leftmost symbol #.

It should be clear that by introducing appropriate states, the Turing machine N can be constructed.

4.4 Non-deterministic Turing Machine

A non-deterministic Turing Machine (NDTM) can be defined in a similar manner as a non-deterministic finite automaton or a non-deterministic pushdown automaton. The transition function of a single tape NDTM will be of the form

δ : Q × Γ → P(Q × Γ × {L, R, N}).

View the computation of a NDTM as a tree. If one of the branches lead us to the accepting state, then the machine accepts the input. It turns out that for every NDTM there is an equivalent Deterministic Turing Machine (DTM). In other words, NDTM and DTM have the same power.

The following simulation leads to the sketch of the proof of the above statement. View the computation of the given NDTM N as a tree rooted at the start state. Each branch of the tree represents a branch of nondeterminism. An equivalent DTM M explores this tree in a breadth first search manner. All branches of the same depth are explored before exploring the branches of successive depth. Due to the nature of the search it is obvious that D will visit every node in the computation tree till it finds an accepting configuration. This can be realized concretely by defining M precisely; we leave the details as an exercise (fairly technical) and conclude with the following theorem.

Theorem 4.4.1 Every non-deterministic Turing Machine has an equivalent deterministic Turing Machine.

4.5 The Church-Turing Thesis

You have some intuitive notion of what an algorithm is. This notion will probably be something like “an algorithm is a procedure consisting of computation steps that can be specified in a finite amount of text”. For example, any “computational process” that can be specified by a Java program, should be considered an algorithm. Similarly, a Turing machine specifies a “computational process” and, therefore, should be considered an algorithm. This leads to the question of whether it is possible to give a mathematical definition of an “algorithm”. We just saw that every Java program represents an algorithm, and that every Turing machine also represents an algorithm. Are these two notions of an algorithm equivalent? The answer is “yes”. In fact, the following theorem states that many different notions of “computational process” are equivalent. (We hope that you have gained sufficient intuition, so that none of the claims in this theorem comes as a surprise to you.)

Theorem 4.5.1 The following computation models are equivalent, i.e., any one of them can be converted to any other one:

1. One-tape Turing machines.

2. k-tape Turing machines, for any k ≥ 1.

3. Non-deterministic Turing machines.

4. Java programs.

5. C++ programs.

6. Lisp programs.

In other words, if we define the notion of an algorithm using any of the models in this theorem, then it does not matter which model we take: All these models give the same notion of an algorithm.

The problem of defining the notion of an algorithm goes back to David Hilbert. On August 8, 1900, at the Second International Congress of Mathematicians in Paris, Hilbert presented a list of problems that he considered crucial for the further development of mathematics. Hilbert’s 10th problem is the following:

Does there exist a finite process that decides whether or not any given polynomial with integer coefficients has integral roots?

Of course, in our language, Hilbert asked whether or not there exists an algorithm that, when given an arbitrary polynomial equation (with integer coefficients), such as

12x^{3}y^{7}z^{5} + 7x^{2}y^{4}z − x^{4} + y^{2}z^{7} − z^{3} + 10 = 0,

has a solution in integers. In 1970, Matiyasevich proved that such an algorithm does not exist. Of course, in order to prove this claim, we first have to agree on what an algorithm is. In the beginning of the twentieth century, mathematicians gave several definitions, such as Turing machines (1936) and the λ-calculus (1936), and they proved that all these are equivalent. Later, after programming languages were invented, it was shown that these older notions of an algorithm are equivalent to notions of an algorithm that are based on C++ programs, Java programs, Lisp program, Pascal programs, etc.

In other words, all attempts to give a rigorous definition of the notion of an algorithm led to the same concept. Because of this, computer scientists nowadays agree on what is called the Church-Turing Thesis:

Church-Turing Thesis: Every computational process that is intuitively considered to be an algorithm can be converted to a Turing machine.

In other words, this basically states that we define an algorithm to be a Turing machine. At this point, you should ask yourself, whether the Church-Turing Thesis can be proved. Alternatively, what has to be done in order to disprove this thesis?

4.6 Exercises

4.1 Construct a Turing machine with one tape, that accepts the language

{0^{2n}1^{n} : n ≥ 0}.

Assume that, at the start of the computation, the tape head is on the leftmost symbol of the input string.

4.2 Construct a Turing machine with one tape, that accepts the language

{w : w contains twice as many 0s as 1s}.

Assume that, at the start of the computation, the tape head is on the leftmost symbol of the input string.

4.3 Let A be the language

A = { w ∈ {a, b, c}^{∗} : w contains more bs than as, and

w contains more cs than as }.

Give an informal description (in plain English) of a Turing machine with one tape, that accepts the language A.

4.4 Construct a Turing machine with one tape, that receives as input a nonnegative integer x, and that returns as output the integer x + 1. Integers are represented as binary strings.

Start of the computation: The tape contains the binary representation of the input x. The tape head is on the leftmost symbol, and the Turing machine is in the start state q_{0}. For example, if x = 431, the tape looks as follows:

… □ □ □ 1 1 0 1 0 1 1 1 1 □ □ □ …^{}

End of the computation: The tape contains the binary representation of the integer x + 1. The tape head is on the leftmost symbol, and the Turing machine is in the final state q_{1}. For our example, the tape looks as follows:

… □ □ □ 1 1 0 1 1 0 0 0 0 □ □ □ …^{}

The Turing machine in this exercise does not have an accept state or a reject state; instead, it has a final state q_{1}. As soon as state q_{1} is entered, the Turing machine terminates. At termination, the contents of the tape is the output of the Turing machine.

4.5 Construct a Turing machine with three tapes, that receives as input two non-negative integers x and y, and that returns as output the integer x + y. Integers are represented as binary strings.

Start of the computation: The first tape contains the binary representation of x, and its head is on the rightmost symbol of x. The second tape contains the binary representation of y, and its head is on the rightmost bit of y. The third tape is empty (that is, contains only □s), and its head is at an arbitrary position. At the start, the Turing machine is in the start state q_{0}.

End of the computation: The first two tapes are empty, and the third tape contains the binary representation of the integer x + y. The head of the third tape is on the rightmost bit of x + y. The Turing machine is in the final state q_{1}.

4.6 Give an informal description (in plain English) of a Turing machine with one tape, that receives as input two non-negative integers x and y, and that returns as output the integer x+y. Integers are represented as binary strings. If you are an adventurous student, you may give a formal definition of your Turing machine.

4.7 Construct a Turing machine with one tape, that receives as input an integer x ≥ 1, and that returns as output the integer x − 1. Integers are represented in binary.

Start of the computation: The tape contains the binary representation of the input x. The tape head is on the leftmost symbol of x, and the Turing machine is in the start state q_{0}.

End of the computation: The tape contains the binary representation of the integer x − 1. The tape head is on the leftmost bit of x − 1, and the Turing machine is in the final state q_{1}.

4.8 Construct a Turing machine with one tape, that receives as input a binary string b.

Start of the computation: The tape head is on the leftmost symbol of b, and the Turing machine is in the start state q_{0}.

End of the computation: The tape contains the binary string bb, i.e., the concatenation of binary string b with itself.

4.9 Show that every language accepted by a single tape read-only Turing Machine is regular. (We are not allowed to write onto the tape.) Show that such a Turing Machine can be simulated by a NFA.

134CHAPTER 4. TURING MACHINES AND THE CHURCH-TURING THESIS

Decidable and Undecidable Languages

We have seen in Chapter 4 that Turing machines form a model for “everything that is intuitively computable”. In this chapter, we consider the limitations of Turing machines. That is, we ask ourselves the question whether or not “everything” is computable. As we will see, the answer is “no”. In fact, we will even see that “most” problems are not solvable by Turing machines and, therefore, not solvable by computers.

5.1 Decidability and enumerability

In Chapter 4, we have defined when a Turing machine accepts an input string, and when it rejects an input string. Based on this, we define two classes of languages.

Definition 5.1.1 Let Σ be an alphabet, and let A ⊆ Σ^{∗} be a language. We say that A is decidable, if there exists a Turing machine M, such that for every string w ∈ Σ^{∗}, the following holds:

1. If w ∈ A, then the computation of the Turing machine M, on the input string w, terminates in the accept state.

2. If w ∉ A, then the computation of the Turing machine M, on the input string w, terminates in the reject state.

136 CHAPTER 5. DECIDABLE AND UNDECIDABLE LANGUAGES

In other words, the language A is decidable, if there exists an algorithm that (i) terminates on every input string w, and (ii) correctly tells us whether w ∈ A or w ∉ A.

Definition 5.1.2 Let Σ be an alphabet, and let A ⊆ Σ^{∗} be a language. We say that A is enumerable, if there exists a Turing machine M, such that for every string w ∈ Σ^{∗}, the following holds:

1. If w ∈ A, then the computation of the Turing machine M, on the input string w, terminates in the accept state.

2. If w ∉ A, then the computation of the Turing machine M, on the input string w, does not terminate in the accept state. That is, the computation terminates in the reject state or the computation does not terminate.

In other words, the language A is enumerable, if there exists an algorithm having the following property. If w ∈ A, then the algorithm terminates on the input string w, and tells us that w ∈ A. On the other hand, if w ∉ A, then (i) either the algorithm terminates on the input string w, and tells us that w ∉ A, or (ii) the algorithm does not terminate on the input string w, in which case it does not tell us that w ∉ A.

The following theorem follows immediately from the above two definitions.

Theorem 5.1.3 Every decidable language is enumerable.

5.2 Examples

5.2.1 Hilbert’s problem

We have seen Hilbert’s problem in Section 4.5: Is there an algorithm that decides, for any given polynomial p with integer coefficients, whether or not p has integral roots? If we formulate this problem in terms of languages, then Hilbert asked whether or not the language

Hilbert = {⟨p⟩ : p is a polynomial with integer coefficients that has an integral root}

is decidable. Here, ⟨p⟩ denotes the binary string that forms an encoding of the polynomial p.

As we mentioned in Section 4.5, it was proven by Matiyasevich in 1970 that the language Hilbert is not decidable. We claim, that this language is enumerable. In order to prove this claim, we have to construct an algorithm P with the following property: For any input polynomial p with integer coefficients,

• if p has an integral root, then algorithm P will find one in a finite amount of time,

• if p does not have an integral root, then either algorithm P terminates and tells us that p does not have an integral root, or algorithm P does not terminate.

Recall that ℤ denotes the set of integers. Algorithm P does the following, on any input polynomial p with integer coefficients. Let n denote the number of variables in p. Algorithm P tries all elements (x_{1}, x_{2}, …, x_{n}) ∈ ℤ^{n}, in a systematic way, and for each such element, it computes p(x_{1}, x_{2}, …, x_{n}). If this value is zero, then algorithm P terminates and accepts the input.

We observe the following:

• If p ∈ Hilbert, then algorithm P terminates and accepts p, provided we are able to visit all elements (x_{1}, x_{2}, …, x_{n}) ∈ ℤ^{n} in a “systematic way”.

• If p ∉ Hilbert, then p(x_{1}, x_{2}, …, x_{n}) ≠ 0 for all (x_{1}, x_{2}, …, x_{n}) ∈ ℤ^{n} and, therefore, algorithm P does not terminate.

But these are exactly the requirements for the language Hilbert to be enumerable.

It remains to explain how we visit all elements (x_{1}, x_{2}, …, x_{n}) ∈ ℤ^{n} in a systematic way. For any integer d ≥ 0, let H_{d} denote the hypercube

H_{d} = [−d, d]^{n}

in ℤ^{n}. That is, H_{d} consists of the set of all points x = (x_{1}, x_{2}, …, x_{n}) ∈ ℤ^{n}, such that there exists at least one index i for which x_{i} = d or x_{i} = −d. We observe that H_{d} contains a finite number of elements. In fact, this number is less than (2d+1)^{n}. The algorithm will visit all elements (x_{1}, x_{2}, …, x_{n}) ∈ ℤ^{n}, in the following order: First, it visits the origin, which is the only element in H_{0}. Then it visits all elements of H_{1}, followed by all elements of H_{2}, etc., etc.

To summarize, we obtain the following algorithm, proving that the language Hilbert is enumerable:

Algorithm Hilbert(⟨p⟩)

n := the number of variables in p;

d := 0; while d ≥ 0 do for each (x_{1}, x_{2}, …, x_{n}) ∈ H_{d} do R := p(x_{1}, x_{2}, …, x_{n});

if R = 0

then terminate and accept endif endfor; d := d + 1

endwhile

Theorem 5.2.1 The language Hilbert is enumerable.

5.2.2 The language A_{DFA}

We define the following language:

A_{DFA} = {⟨M, w⟩ : M is a deterministic finite automaton that accepts the string w}.

Here, ⟨M, w⟩ denotes the binary string that forms an encoding of the finite automaton M and the string w that is given as input to M.

We claim that the language A_{DFA} is decidable. In order to prove this claim, we have to construct an algorithm with the following property, for any given input string u:

• If u is the encoding of a deterministic finite automaton M and a string w, and if M accepts w, then the algorithm terminates in its accept state.

• In all other cases, the algorithm terminates in its reject state.

An algorithm that exactly does this, is easy to obtain: On input u, the algorithm first checks whether or not u encodes a deterministic finite automaton M and a string w. If this is not the case, then it terminates and rejects the input string u. Otherwise, the algorithm “constructs” M and w, and then simulates the computation of M on the input string w. If M accepts w, then the algorithm terminates and accepts the input string u. If M does not accept w, then the algorithm terminates and rejects the input string u.

Theorem 5.2.2 The language A_{DFA} is decidable.

5.2.3 The language A_{NFA}

We define the following language:

A_{NFA} = {⟨M, w⟩ : M is a nondeterministic finite automaton that accepts the string w}.

To prove that this language is decidable, consider the algorithm that does the following: On input u, the algorithm first checks whether or not u encodes a nondeterministic finite automaton M and a string w. If this is not the case, then it terminates and rejects the input string u. Otherwise, the algorithm constructs M and w. Since a computation of M (on input w) is not unique, the algorithm first converts M to an equivalent deterministic finite automaton N. Then, it proceeds as in Section 5.2.2.

Observe that the construction for converting a nondeterministic finite automaton to a deterministic finite automaton (see Section 2.5) is algorithmic, in the sense that it can be described by an algorithm. Because of this, the algorithm described above is a valid algorithm; it accepts all strings u that are in A_{NFA}, and it rejects all strings u that are not in A_{NFA}.

Theorem 5.2.3 The language A_{NFA} is decidable.

5.2.4 The language A_{CFG}

We define the following language:

A_{CFG} = {⟨G, w⟩ : G is a context-free grammar such that w ∈ L(G)}.

We claim that this language is decidable. In order to prove this claim, consider a string u that encodes a context-free grammar G = (V, Σ, S, R) and a string w ∈ Σ^{∗}. Deciding whether or not w ∈ L(G) is equivalent to deciding whether or not S ⇒^{∗} w. A first idea to decide this is by trying all possible derivations that start with the start variable S, and that use rules of R. The problem is that, in case w ∉ L(G), it is not clear how many such derivations have to be checked before we can be sure that w is not in the language of G: If w ∈ L(G), then it may be that w can only be derived from S, by first deriving a very long string, say v, and then use rules to shorten it so as to obtain the string w. Since there is no obvious upper bound on the length of the string v, we have to be careful.

The trick is to do the following. First, convert the grammar G to an equivalent grammar G′ in Chomsky normal form. (The construction given in Section 3.4 can be described by an algorithm.) Let n be the length of the string w. Then, if w ∈ L(G) = L(G′), any derivation of w, from the start variable of G′, consists of exactly 2n − 1 steps (where a “step” is defined as applying one rule of G′). Hence, we can decide whether or not w ∈ L(G), by trying all possible derivations, in G′, consisting of 2n − 1 steps. If one of these (finite number of) derivations leads to the string w, then w ∈ L(G). Otherwise, w ∉ L(G).

Theorem 5.2.4 The language A_{CFG} is decidable.

In fact, the arguments above imply the following result:

Theorem 5.2.5 Every context-free language is decidable.

Proof. Let Σ be an alphabet, and let A ⊆ Σ^{∗} be an arbitrary context-free language. There exists a context-free grammar in Chomsky normal form, whose language is equal to A. Given an arbitrary string w ∈ Σ^{∗}, we have seen above how we can decide whether or not w can be derived from the start variable of this grammar.

5.2.5 The language A_{TM}

After having seen the languages A_{DFA}, A_{NFA}, and A_{CFG}, it is natural to consider the following language:

A_{TM} = {⟨M, w⟩ : M is a Turing machine that accepts the string w}.

We will prove in Section 5.5 that this language is not decidable. Let us mention here what this means:

There is no algorithm that, when given an arbitrary algorithm M and an arbitrary input string w for M, decides in a finite amount of time, whether or not M accepts w.

Interestingly, the language A_{TM} is enumerable. In order to prove this, we have to construct an algorithm P with the following property: For any given input string u,

• if u encodes a Turing machine M and an input string w for M, and if M accepts w, then algorithm P terminates in its accept state,

• in all other cases, either algorithm P terminates in its reject state, or algorithm P does not terminate.

On input string u = ⟨M, w⟩, algorithm P does the following:

1. It simulates the computation of M on input w.

2. If M terminates in its accept state, then P terminates in its accept state.

3. If M terminates in its reject state, then P terminates in its reject state.

4. if M does not terminate, then P does not terminate.

Hence, if u = ⟨M, w⟩ ∈ A_{TM} , then M accepts w and, therefore, P accepts u. On the other hand, if u = ⟨M, w⟩ ∉ A_{TM} , then M does not accept w. This means that, on input w, M either terminates in its reject state or does not terminate. But this implies that, on input u, P either terminates in its reject state or does not terminate. This proves that algorithm P has the properties that are needed in order to show that the language A_{TM} is enumerable.

Theorem 5.2.6 The language A_{TM} is enumerable.

5.3 Most languages are not enumerable

In this section, we will prove that there exist languages that are not enumerable. The proof is based on the following two facts:

• The set consisting of all enumerable languages is countable.

• The set consisting of all languages is not countable.

Before we prove these facts, we review the notion of countability.

5.3.1 Countable sets

Let A and B be two sets, and let f : A → B be a function. Recall that f is called a bijection, if

• f is one-to-one (or injective), i.e., for any two distinct elements a and a′ in A, we have f(a) ≠ f(a′), and

• f is onto (or surjective), i.e., for each element b ∈ B, there exists an element a ∈ A, such that f(a) = b.

The set of natural numbers is denoted by ℕ. That is, ℕ = {1, 2, 3, …}.

Definition 5.3.1 Let A and B be two sets. We say that A and B have the same size, if there exists a bijection f : A → B.

Definition 5.3.2 Let A be a set. We say that A is countable, if A is finite, or A and ℕ have the same size.

In other words, if A is an infinite and countable set, then there exists a bijection f : ℕ → A, and we can write A as

A = {f(1), f(2), f(3), f(4), …}.

Since f is a bijection, every element of A occurs exactly once in the set on the right-hand side. This means that we can number the elements of A using the positive integers: Every element of A receives a unique number.

Theorem 5.3.3 The following sets are countable:

1. The set ℤ of integers:

ℤ = {…, −3, −2, −1, 0, 1, 2, 3, …}.

2. The Cartesian product ℕ × ℕ:

ℕ × ℕ = {(m, n) : m ∈ ℕ, n ∈ ℕ}.

3. The set ℚ of rational numbers:

ℚ = {m/n : m ∈ ℤ, n ∈ ℤ, n ≠ 0}.

Proof. To prove that the set ℤ is countable, we have to give each element of ℤ a unique number in ℕ. We obtain this numbering, by listing the elements of ℤ in the following order:

0, 1, −1, 2, −2, 3, −3, 4, −4, …

In this (infinite) list, every element of ℤ occurs exactly once. The number of an element of ℤ is given by its position in this list.

Formally, define the function f : ℕ → ℤ by

f(n) = \{ \begin{matrix}n/2 & if n is even, \\ −(n − 1)/2 & if n is odd. \end{matrix}

This function f is a bijection and, therefore, the sets ℕ and ℤ have the same size. Hence, the set ℤ is countable.

For the proofs of the other two claims, we refer to the course COMP 1805.

Theorem 5.3.4 The set ℝ of real numbers is not countable.

Proof. Define

A = {x ∈ ℝ : 0 ≤ x ≤ 1}.

We will prove that the set A is not countable. This will imply that the set ℝ is not countable, because A ⊆ ℝ.

The proof that A is not countable is by contradiction. So we assume that A is countable. Then there exists a bijection f : ℕ → A. Hence, we can write

A = {f(1), f(2), f(3), …},  (5.1)

where every element of A occurs exactly once in the set on the right-hand side. Hence, for each n ∈ ℕ, f(n) is a real number between zero and one.

Consider the real number f(1). We can write this number in decimal notation as

f(1) = 0.d_{11}d_{12}d_{13} …,

where each d_{1i} is a digit in the set {0, 1, 2, …, 9}. In general, for every n ∈ ℕ, we can write the real number f(n) as

f(n) = 0.d_{n1}d_{n2}d_{n3} …,

where, again, each d_{ni} is a digit in {0, 1, 2, …, 9}.

We define the real number

x = 0.d_{1}d_{2}d_{3} …,

where, for each n ≥ 1,

d_{n} = \{ \begin{matrix}4 & if d_{nn} ≠ 4, \\ 5 & if d_{nn} = 4. \end{matrix}

Observe that x is a real number between zero and one, i.e., x ∈ A. Therefore, by (5.1), there is an element n ∈ ℕ, such that f(n) = x. We compare the n-th digits of f(n) and x:

• The n-th digit of f(n) is equal to d_{nn}.

• The n-th digit of x is equal to d_{n}.

Since f(n) and x are equal, their n-th digits must be equal, i.e., d_{nn} = d_{n}. But, by the definition of d_{n}, we have d_{nn} ≠ d_{n}. This is a contradition and, therefore, the set A is not countable.

5.3.2 The set of enumerable languages is countable

We define the set E as

E = {A : A ⊆ {0, 1}^{∗} is an enumerable language}.

In words, E is the set whose elements are the enumerable languages. Every element of E is an enumerable language. Hence, every element of the set E is itself a set consisting of strings.

Lemma 5.3.5 The set E is countable.

Proof. Let A ⊆ {0, 1}^{∗} be an enumerable language. There exists a Turing machine T_{A}, that satisfies the conditions in Definition 5.1.2. This Turing machine T_{A} can be uniquely specified by a string in English. This string can be converted to a binary string s_{A}. Hence, the binary string s_{A} is a unique encoding of the Turing machine T_{A}.

Consider the set

S = {s_{A} : A ⊆ {0, 1}^{∗} is an enumerable language}.

Observe that the function f : E → S, defined by f(A) = s_{A} for each A ∈ E, is a bijection. Therefore, the sets E and S have the same size. Hence, in order to prove that the set E is countable, it is sufficient to prove that the set S is countable.

Why is the set S countable? For each integer n ≥ 0, there are exactly 2^{n} binary strings of length n. Hence, the set S contains at most 2^{n} strings of length n; in particular, the number of strings in S having length n is finite. Therefore, we obtain an infinite list of the elements of S, in the following way:

• List all strings in S having length 0. (Well, the empty string is not in S, so in this step, nothing happens.)

• List all strings in S having length 1.

• List all strings in S having length 2.

• List all strings in S having length 3.

• Etcetera, etcetera.

In this infinite list, every element of S occurs exactly once. Therefore, S is countable.

5.3.3 The set of all languages is not countable

We define the set L as

L = {A : A ⊆ {0, 1}^{∗} is a language}.

In words, L is the set consisting of all languages. Every element of the set L is a set consisting of strings.

Lemma 5.3.6 The set L is not countable.

Proof. We define the set B as

B = {w : w is an infinite binary sequence}.

We claim that this set is not countable. The proof of this claim is almost identical to the proof of Theorem 5.3.4. We assume that the set B is countable. Then there exists a bijection f : ℕ → B. Hence, we can write

B = {f(1), f(2), f(3), …},  (5.2)

where every element of B occurs exactly once in the set on the right-hand side. Hence, for each n ∈ ℕ, f(n) is an infinite binary sequence.

We define the infinite binary sequence w = w_{1}w_{2}w_{3} …, where, for each n ≥ 1,

w_{n} = \{ \begin{matrix}1 & if the n-th bit of f(n) is 0, \\ 0 & if the n-th bit of f(n) is 1. \end{matrix}

Since w ∈ B, it follows from (5.2) that there is an element n ∈ ℕ, such that f(n) = w. Hence, the n-th bits of f(n) and w are equal. But, by definition, these n-th bits are not equal. This is a contradition and, therefore, the set B is not countable.

In the rest of the proof, we will show that the sets L and B have the same size. Since B is not countable, this will imply that L is not countable.

In order to prove that L and B have the same size, we have to show that there exists a bijection

g : L → B.

We first observe that the set {0, 1}^{∗} is countable, because for each n ≥ 0, there are exactly 2^{n} strings of length n. In fact, we can write

{0, 1}^{∗} = {ϵ, 0, 1, 00, 01, 10, 11, 000, 001, 010, 100, 011, 101, 110, 111, …}.

For each n ≥ 1, we denote by s_{n} the n-th string in this list. Hence,

{0, 1}^{∗} = {s_{1}, s_{2}, s_{3}, …}.  (5.3)

Now we are ready to define the bijection g : L → B: Let A ∈ L, i.e., A ⊆ {0, 1}^{∗} is a language. We define the infinite binary sequence g(A) as follows: For each n ≥ 1, the n-th bit of g(A) is equal to

\{ \begin{matrix}1 & if s_{n} ∈ A, \\ 0 & if s_{n} ∉ A. \end{matrix}

In words, the infinite binary sequence g(A) contains a one exactly in those positions n for which the string s_{n} in (5.3) is in the language A.

To give an example, assume that A is the language consisting of all binary strings that start with 0. The following table gives the corresponding infinite binary sequence g(A) (this sequence is obtained by reading the rightmost column from top to bottom):

{0, 1}^{∗}  A  g(A)
ϵ  not in A 0
0 in A 1
1  not in A 0
00  in A  1
01  in A  1
10 not in A 0
11 not in A 0
000  in A  1
001  in A  1
010  in A  1
100 not in A 0
011 in A 1
101 not in A 0
110 not in A 0
111 not in A 0
⋮  ⋮  ⋮

The function g defined above has the following properties:

• If A and A′ are two different languages in L, then g(A) ≠ g(A′).

• For every infinite binary sequence w in B, there exists a language A in L, such that g(A) = w.

This means that the function g is a bijection from L to B.

5.3.4 There are languages that are not enumerable

We have proved that the set

E = {A : A ⊆ {0, 1}^{∗} is an enumerable language}

is countable, whereas the set

L = {A : A ⊆ {0, 1}^{∗} is a language}

is not countable. This means that there are “more” languages in L than there are in E, proving the following result:

Theorem 5.3.7 There exist languages that are not enumerable.

The proof given above shows the existence of languages that are not enumerable. Since every decidable language is enumerable (see Theorem 5.1.3), it also shows the existence of languages that are not decidable. However, the proof does not give us a specific example of a language that is not enumerable, or that is not decidable. In the next sections, we will see examples of such languages.

5.4 The Halting Problem

We define the following language:

Halt = {⟨P, w⟩ : P is a Java program that terminates on the input string w}.

In Section 5.2.5, we proved that the language A_{TM} is enumerable. Using a simple modification of this proof, it can be shown that the language Halt is enumerable.

Theorem 5.4.1 The language Halt is undecidable.

Proof. The proof is by contradiction. So we assume that the language Halt is decidable. Then there exists a Java program H, that takes as input a string of the form ⟨P, w⟩, where P is an arbitrary Java program, and w is an arbitrary input for P. The program H has the following property:

• H outputs true, if ⟨P, w⟩ ∈ Halt (i.e., program P terminates on input w).

• H outputs false, if ⟨P, w⟩ ∉ Halt, (i.e., program P does not terminate on input w).

We will write the output of H as H(P, w). Moreover, we will denote by P (w) the computation obtained by running the program P on the input w. Hence,

H(P, w) = \{ \begin{matrix}true & if P (w) terminates, \\ false & if P (w) does not terminate. \end{matrix}

Consider the following algorithm Q, which takes as input the encoding ⟨P ⟩ of an arbitrary Java program P:

Algorithm Q(⟨P ⟩):

while H(P, ⟨P ⟩) = true do have a beer endwhile

Since H is a Java program, this new algorithm Q can also be written as a Java program. Observe that

Q(⟨P ⟩) terminates if and only if H(P, ⟨P ⟩) = false.

This means that for every Java program P,

Q(⟨P ⟩) terminates if and only if P (⟨P ⟩) does not terminate.  (5.4)

What happens if we run the Java program Q on the input string ⟨Q⟩? In other words, what happens if we run Q(⟨Q⟩)? Then, in (5.4), we have to replace P by Q. Hence,

Q(⟨Q⟩) terminates if and only if Q(⟨Q⟩) does not terminate.

This is obviously a contradiction, and we can conclude that the Java program H does not exist. Hence, the language Halt is undecidable.

Remark 5.4.2 In this proof, we run the Java program Q on the input ⟨Q⟩. This means that the input to Q is a description of itself. In other words, we give Q itself as input. This is an example of what is called self-reference. If this does not confuse you yet, consider the following statement S:

This statement is false.

Is the statement S true or false? Another example of self-reference can be found in Remark 5.4.2 of the lecture notes Discrete Structures II by A. Maheshwari and M. Smid.

5.5 The language A_{TM} is undecidable

Recall the definition of the language A_{TM} :

A_{TM} = {⟨M, w⟩ : M is a Turing machine that accepts the string w}.

We know that this language is enumerable; see Theorem 5.2.6. In this section, we prove that A_{TM} is undecidable.

The proof is by contradiction. So we assume that A_{TM} is decidable. Then there exists a Turing machine H that has the following property: On every input string ⟨M, w⟩,

• H terminates in its accept state, if ⟨M, w⟩ ∈ A_{TM} (i.e., if M accepts w),

• H terminates in its reject state, if ⟨M, w⟩ ∉ A_{TM} (i.e., if M rejects w or M does not terminate on input w).

Observe that H terminates on any input string ⟨M, w⟩.

We construct a new Turing machine D, that does the following: On input ⟨M⟩, the Turing machine D uses H as a subroutine to determine what M does when it is given its own description as input. Once D has determined this information, it does the opposite from what H does.

Turing machine D: On input ⟨M⟩, where M is a Turing machine, the new Turing machine D does the following:

Step 1: Run the Turing machine H on the input ⟨M, ⟨M⟩⟩.

Step 2:

• If H terminates in its accept state, then D terminates in its reject state.

• If H terminates in its reject state, then D terminates in its accept state.

First, observe that this new Turing machine D terminates on every input string ⟨M⟩, because H terminates on every input. Next, observe that, for every possible input string ⟨M⟩,

• D terminates in its accept state, if ⟨M, ⟨M⟩⟩ ∉ A_{TM} (i.e., if M rejects ⟨M⟩ or M does not terminate on input ⟨M⟩), 5.6. THE RELATION BETWEEN DECIDABLE AND ENUMERABLE LANGUAGES151

• D terminates in its reject state, if ⟨M, ⟨M⟩⟩ ∈ A_{TM} (i.e., if M accepts

⟨M⟩). This means that for every string ⟨M⟩,

• D accepts ⟨M⟩, if M rejects ⟨M⟩ or M does not terminate on input ⟨M⟩,

• D rejects ⟨M⟩, if M accepts ⟨M⟩.

We now consider what happens if we give the Turing machine D the string ⟨D⟩ as input, i.e., we take M = D:

• D accepts ⟨D⟩, if D rejects ⟨D⟩ or D does not terminate on input ⟨D⟩,

• D rejects ⟨D⟩, if D accepts ⟨D⟩. Since D terminates on every input string, this means that

• D accepts ⟨D⟩, if D rejects ⟨D⟩,

• D rejects ⟨D⟩, if D accepts ⟨D⟩. This is clearly a contradiction. Therefore, the Turing machine H that decides the language A_{TM} cannot exist, and A_{TM} is not decidable. Theorem 5.5.1 The language A_{TM} is undecidable.

5.6 The relation between decidable and enu-

merable languages

We know from Theorem 5.1.3 that every decidable language is enumerable. On the other hand, we know from Theorems 5.2.6 and 5.5.1 that the converse is not true. The following result should not come as a surprise: Theorem 5.6.1 Let Σ be an alphabet, and let A ⊆ Σ^{∗} be a language. Then, A is decidable if and only if both A and its complement \overline{A} are enumerable.

Proof. We first assume that A is decidable. Then, by Theorem 5.1.3, A is enumerable. Since A is decidable, it is not difficult to see that \overline{A} is also decidable. Then, again by Theorem 5.1.3, \overline{A} is enumerable.

To prove the converse, we assume that both A and \overline{A} are enumerable. Since A is enumerable, there exists a Turing machine M_{1}, such that for any string w ∈ Σ^{∗}, the following holds:

• If w ∈ A, then the computation of M_{1}, on the input string w, terminates in the accept state of M_{1}.

• If w ∉ A, then the computation of M_{1}, on the input string w, terminates in the reject state of M_{1} or does not terminate.

Similarly, since \overline{A} is enumerable, there exists a Turing machine M_{2}, such that for any string w ∈ Σ^{∗}, the following holds:

• If w ∈ \overline{A}, then the computation of M_{2}, on the input string w, terminates in the accept state of M_{2}.

• If w ∉ \overline{A}, then the computation of M_{2}, on the input string w, terminates in the reject state of M_{2} or does not terminate.

We construct a two-tape Turing machine M:

Two-tape Turing machine M: For any input string w ∈ Σ^{∗}, M does the following:

• M simulates the computation of M_{1}, on input w, on the first tape, and, simultaneously, it simulates the computation of M_{2}, on input w, on the second tape.

• If the simulation of M_{1} terminates in the accept state of M_{1}, then M terminates in its accept state.

• If the simulation of M_{2} terminates in the accept state of M_{2}, then M terminates in its reject state.

Observe the following:

• If w ∈ A, then M_{1} terminates in its accept state and, therefore, M terminates in its accept state.

• If w ∉ A, then M_{2} terminates in its accept state and, therefore, M terminates in its reject state.

We conclude that the Turing machine M accepts all strings in A, and rejects all strings that are not in A. This proves that the language A is decidable.

Theorem 5.6.2 The language \overline{A_{TM}} is not enumerable.

Proof. We know from Theorems 5.2.6 and 5.5.1 that the language A_{TM} is enumerable but not decidable. Combining these facts with Theorem 5.6.1 implies that the language \overline{A_{TM}} is not enumerable.

5.7 Exercises

5.1 Prove that the language Halt, see Section 5.4, is enumerable.

5.2 Prove that the language

{w ∈ {0, 1}^{∗} : w is the binary representation of 2^{n}, for some n ≥ 0}

is decidable. In other words, construct a Turing machine that gets as input an arbitrary number x ∈ ℕ, represented in binary as a string w, and that decides whether or not x is a power of two.

5.3 Let F be the set of all functions f : ℕ → ℕ. Prove that F is not countable.

5.4 A function f : ℕ → ℕ is called computable, if there exists a Turing machine, that gets as input an arbitrary positive integer n, written in binary, and gives as output the value of f(n), again written in binary. This Turing machine has a final state. As soon as the Turing machine enters this final state, the computation terminates, and the output is the binary string that is written on its tape.

Prove that there exist functions f : ℕ → ℕ that are not computable.

5.5 Let n be a fixed positive integer, and let k be the number of bits in the binary representation of n. (Hence, k = 1 + ⌊log n⌋.) Construct a Turing machine with one tape, tape alphabet {0, 1, □}, and exactly k + 1 states q_{0}, q_{1}, …, q_{k}, that does the following:

Start of the computation: The tape is empty, i.e., every cell of the tape contains □, and the Turing machine is in the start state q_{0}.

End of the computation: The tape contains the binary representation of the integer n, the tape head is on the rightmost bit of the binary representation of n, and the Turing machine is in the final state q_{k}.

The Turing machine in this exercise does not have an accept state or a reject state; instead, it has a final state q_{k}. As soon as state q_{k} is entered, the Turing machine terminates.

5.6 Give an informal description (in plain English) of a Turing machine with three tapes, that gets as input the binary representation of an arbitrary integer m ≥ 1, and returns as output the unary representation of m.

Start of the computation: The first tape contains the binary representation of the input m. The other two tapes are empty (i.e., contain only □s). The Turing machine is in the start state.

End of the computation: The third tape contains the unary representation of m, i.e., a string consisting of m many ones. The Turing machine is in the final state.

The Turing machine in this exercise does not have an accept state or a reject state; instead, it has a final state. As soon as this final state is entered, the Turing machine terminates.

Hint: Use the second tape to maintain a string of ones, whose length is a power of two.

5.7 In this exercise, you are asked to prove that the busy beaver function BB : ℕ → ℕ is not computable.

For any integer n ≥ 1, we define TM_{n} to be the set of all Turing machines M, such that

• M has one tape,

• M has exactly n states, • the tape alphabet of M is {0, 1, □}, and

• M terminates, when given the empty string ϵ as input.

For every Turing machine M ∈ TM_{n}, we define f(M) to be the number of ones on the tape, after the computation of M, on the empty input string, has terminated.

The busy beaver function BB : ℕ → ℕ is defined as

BB(n) := max{f(M) : M ∈ TM_{n}}, for every n ≥ 1.

In words, BB(n) is the maximum number of ones that any Turing machine with n states can produce, when given the empty string as input, and assuming the Turing machine terminates on this input.

Prove that the function BB is not computable.

Hint: Assume that BB is computable. Then there exists a Turing machine M that, for any given n ≥ 1, computes the value of BB(n). Fix a large integer n ≥ 1. Define (in plain English) a Turing machine that, when given the empty string as input, terminates and outputs a string consisting of more than BB(n) many ones. Use Exercises 5.5 and 5.6 to argue that there exists such a Turing machine having O(log n) states. Then, if you assume that n is large enough, the number of states is at most n.

5.8 We define the following language:

L = {u : u = ⟨0, M, w⟩ for some ⟨M, w⟩ ∈ A_{TM} ,
or u = ⟨1, M, w⟩ for some ⟨M, w⟩ ∉ A_{TM} } .

Prove that both J and its complement \overline{J} are not enumerable.

Hint: There are two ways to solve this exercise. In the first solution, (i) you assume that J is enumerable, and then prove that A_{TM} is decidable, and (ii) you assume that \overline{J} is enumerable, and then prove that A_{TM} is decidable. In the second solution, (i) you assume that J is enumerable, and then prove that \overline{A_{TM}} is enumerable, and (ii) you assume that \overline{J} is enumerable, and then prove that \overline{A_{TM}} is enumerable.

156 CHAPTER 5. DECIDABLE AND UNDECIDABLE LANGUAGES

Complexity Theory

In the previous chapters, we have considered the problem of what can be computed by Turing machines (i.e., computers) and what cannot be computed. We did not, however, take the efficiency of the computations into account. In this chapter, we introduce a classification of decidable languages A, based on the running time of the “best” algorithm that decides A. That is, given a decidable language A, we are interested in the “fastest” algorithm that, for any given string w, decides whether or not w ∈ A.

6.1 The running time of algorithms

Let M be a Turing machine, and let w be an input string for M. We define the running time t_{M}(w) of M on input w as

t_{M}(w) := the number of computation steps made by M on input w.

As usual, we denote by |w|, the number of symbols in the string w. We denote the set of non-negative integers by ℕ_{0}.

Definition 6.1.1 Let Σ be an alphabet, let T : ℕ_{0} → ℕ_{0} be a function, let A ⊆ Σ^{∗} be a decidable language, and let F : Σ^{∗} → Σ^{∗} be a computable function.

• We say that the Turing machine M decides the language A in time T,

if

t_{M}(w) ≤ T (|w|)

for all strings w in Σ^{∗}.

• We say that the Turing machine M computes the function F in time

T, if

t_{M}(w) ≤ T (|w|)

for all strings w ∈ Σ^{∗}.

In other words, the “running time function” T is a function of the length of the input, which we usually denote by n. For any n, the value of T (n) is an upper bound on the running time of the Turing machine M, on any input string of length n.

To give an example, consider the Turing machine of Section 4.2.1 that decides, using one tape, the language consisting of all palindromes. The tape head of this Turing machine moves from the left to the right, then back to the left, then to the right again, back to the left, etc. Each time it reaches the leftmost or rightmost symbol, it deletes this symbol. The running time of this Turing machine, on any input string of length n, is

O(1 + 2 + 3 + … + n) = O(n^{2}).

On the other hand, the running time of the Turing machine of Section 4.2.2, which also decides the palindromes, but using two tapes instead of just one, is O(n).

In Section 4.5, we mentioned that all computation models listed there are equivalent, in the sense that if a language can be decided in one model, it can be decided in any of the other models. We just saw, however, that the language consisting of all palindromes allows a faster algorithm on a twotape Turing machine than on one-tape Turing machines. (Even though we did not prove this, it is true that Ω(n^{2}) is a lower bound on the running time to decide palindromes on a one-tape Turing machine.) The following theorem can be proved.

Theorem 6.1.2 Let A be a language (resp. let F be a function) that can be decided (resp. computed) in time T by an algorithm of type M. Then there is an algorithm of type N that decides A (resp. computes F) in time T′, where

M  N  T′
k-tape Turing machine one-tape Turing machine O(T^{2})
one-tape Turing machine Java program  O(T^{2})
Java program  k-tape Turing machine O(T^{4})

6.2 The complexity class P

Definition 6.2.1 We say that algorithm M decides the language A (resp. computes the function F) in polynomial time, if there exists an integer k ≥ 1, such that the running time of M is O(n^{k}), for any input string of length n.

It follows from Theorem 6.1.2 that this notion of “polynomial time” does not depend on the model of computation:

Theorem 6.2.2 Consider the models of computation “Java program”, “ktape Turing machine”, and “one-tape Turing machine”. If a language can be decided (resp. a function can be computed) in polynomial time in one of these models, then it can be decided (resp. computed) in polynomial time in all of these models.

Because of this theorem, we can define the following two complexity classes:

P := {A : the language A is decidable in polynomial time},

and

FP := {F : the function F is computable in polynomial time}.

6.2.1 Some examples

Palindromes

Let Pal be the language

Pal := {w ∈ {a, b}^{∗} : w is a palindrome}.

We have seen that there exists a one-tape Turing machine that decides Pal in O(n^{2}) time. Therefore, Pal ∈ P.

Some functions in FP

The following functions are in the class FP:

• F_{1} : ℕ_{0} → ℕ_{0} defined by F_{1}(x) := x + 1,

• F_{2} : ℕ_{0}^{2} → ℕ_{0} defined by F_{2}(x, y) := x + y,

• F_{3} : ℕ_{0}^{2} → ℕ_{0} defined by F_{3}(x, y) := xy.

Context-free languages

We have shown in Section 5.2.4 that every context-free language is decidable. The algorithm presented there, however, does not run in polynomial time. In COMP 3804, the following result will be shown (using a technique called dynamic programming):

Theorem 6.2.3 Let Σ be an alphabet, and let A ⊆ Σ^{∗} be a context-free language. Then A ∈ P.

Observe that, obviously, every language in P is decidable.

The 2-coloring problem

Let G be a graph with vertex set V and edge set E. We say that G is 2-colorable, if it is possible to give each vertex of V a color such that

1. for each edge (u, v) ∈ E, the vertices u and v have different colors, and

2. only two colors are used to color all vertices.

See Figure 6.1 for an example. We define the following language:

2Color := {⟨G⟩ : the graph G is 2-colorable},

where ⟨G⟩ denotes the binary string that encodes the graph G.

We claim that 2Color ∈ P. In order to show this, we have to construct an algorithm that decides in polynomial time, whether or not any given graph is 2-colorable.

Let G be an arbitrary graph with vertex set V = {1, 2, …, m}. The edge set of G is given by an adjacency matrix. This matrix, which we denote by E, is a two-dimensional array with m rows and m columns. For all i and j with 1 ≤ i ≤ m and 1 ≤ j ≤ m, we have

E(i, j) = \{ \begin{matrix}1 & if (i, j) is an edge of G, \\ 0 & otherwise. \end{matrix}

The length of the input G, i.e., the number of bits needed to specify G, is equal to m^{2} =: n. We will present an algorithm that decides, in O(n) time, whether or not the graph G is 2-colorable.

r  b b b  r r  b G_{1}  G_{2}

Figure 6.1: The graph G_{1} is 2-colorable; r stands for red; b stands for blue. The graph G_{2} is not 2-colorable.

The algorithm uses the colors red and blue. It gives the first vertex the color red. Then, the algorithm considers all vertices that are connected by an edge to the first vertex, and colors them blue. Now the algorithm is done with the first vertex; it marks this first vertex.

Next, the algorithm chooses a vertex i that already has a color, but that has not been marked. Then it considers all vertices j that are connected by an edge to i. If j has the same color as i, then the input graph G is not 2-colorable. Otherwise, if vertex j does not have a color yet, the algorithm gives j the color that is different from i’s color. After having done this for all neighbors j of i, the algorithm is done with vertex i, so it marks i.

It may happen that there is no vertex i that already has a color but that has not been marked. (In other words, each vertex i that is not marked does not have a color yet.) In this case, the algorithm chooses an arbitrary vertex i having this property, and colors it red. (This vertex i is the first vertex in its connected component that gets a color.)

This procedure is repeated until all vertices of G have been marked.

We now give a formal description of this algorithm. Vertex i has been marked, if

1. i has a color,

2. all vertices that are connected by an edge to i have a color, and 3. the algorithm has verified that each vertex that is connected by an edge

to i has a color different from i’s color.

The algorithm uses two arrays f(1 … m) and a(1 … m), and a variable M. The value of f(i) is equal to the color (red or blue) of vertex i; if i does not have a color yet, then f(i) = 0. The value of a(i) is equal to

a(i) = \{ \begin{matrix}1 & if vertex i has been marked, \\ 0 & otherwise. \end{matrix}

The value of M is equal to the number of marked vertices. The algorithm is presented in Figure 6.2. You are encouraged to convince yourself of the correctness of this algorithm. That is, you should convince yourself that this algorithm returns YES if the graph G is 2-colorable, whereas it returns NO otherwise.

What is the running time of this algorithm? First we count the number of iterations of the outer while-loop. In one iteration, either M increases by one, or a vertex i, for which a(i) = 0, gets the color red. In the latter case, the variable M is increased during the next iteration of the outer while-loop. Since, during the entire outer while-loop, the value of M is increased from zero to m, it follows that there are at most 2m iterations of the outer whileloop. (In fact, the number of iterations is equal to m plus the number of connected components of G minus one.)

One iteration of the outer while-loop takes O(m) time. Hence, the total running time of the algorithm is O(m^{2}), which is O(n). Therefore, we have shown that 2Color ∈ P.

6.3 The complexity class NP

Before we define the class NP, we consider some examples.

Example 6.3.1 Let G be a graph with vertex set V and edge set E, and let k ≥ 1 be an integer. We say that G is k-colorable, if it is possible to give each vertex of V a color such that

1. for each edge (u, v) ∈ E, the vertices u and v have different colors, and

2. at most k different colors are used to color all vertices.

6.3. THE COMPLEXITY CLASS NP 163

Algorithm 2Color for i := 1 to m do f(i) := 0; a(i) := 0 endfor; f(1) := red; M := 0; while M ≠ m do (∗ Find the minimum index i for which vertex i has not

been marked, but has a color already ∗) bool := false; i := 1; while bool = false and i ≤ m do if a(i) = 0 and f(i) ≠ 0 then bool := true else i := i + 1 endif; endwhile; (∗ If bool = true, then i is the smallest index such that

a(i) = 0 and f(i) ≠ 0.

If bool = false, then for all i, the following holds: if a(i) = 0, then f(i) = 0; because M < m, there is at least one such i. ∗)

if bool = true then for j := 1 to m

do if E(i, j) = 1

then if f(i) = f(j)

then return NO and terminate else if f(j) = 0

then if f(i) = red then f(j) := blue else f(j) := red endif

endif endif

endif endfor; a(i) := 1; M := M + 1;

else i := 1; while a(i) ≠ 0 do i := i + 1 endwhile; (∗ an unvisited connected component starts at vertex i ∗) f(i) := red

endif endwhile; return YES

Figure 6.2: An algorithm that decides whether or not a graph G is 2colorable.

We define the following language:

kColor := {⟨G⟩ : the graph G is k-colorable}.

We have seen that for k = 2, this problem is in the class P. For k ≥ 3, it is not known whether there exists an algorithm that decides, in polynomial time, whether or not any given graph is k-colorable. In other words, for k ≥ 3, it is not known whether or not kColor is in the class P.

Example 6.3.2 Let G be a graph with vertex set V = {1, 2, …, m} and edge set E. A Hamilton cycle is a cycle in G that visits each vertex exactly once. Formally, it is a sequence v_{1}, v_{2}, …, v_{m} of vertices such that

1. {v_{1}, v_{2}, …, v_{m}} = V, and

2. {(v_{1}, v_{2}), (v_{2}, v_{3}), …, (v_{m−1}, v_{m}), (v_{m}, v_{1})} ⊆ E.

We define the following language:

HC := {⟨G⟩ : the graph G contains a Hamilton cycle}.

It is not known whether or not HC is in the class P.

Example 6.3.3 The sum of subset language is defined as follows:

\begin{aligned}SOS := \{⟨a_{1}, a_{2}, …, a_{m}, b⟩ : & m, a_{1}, a_{2}, …, a_{m}, b ∈ ℕ_{0} and \\ & ∃I ⊆ \{1, 2, …, m\}, ∑_{i∈I} a_{i} = b\}. \end{aligned}

Also in this case, no polynomial-time algorithm is known that decides the language SOS. That is, it is not known whether or not SOS is in the class P.

Example 6.3.4 An integer x ≥ 2 is a prime number, if there are no a, b ∈ ℕ such that a ≠ x, b ≠ x, and x = ab. Hence, the language of all non-primes that are greater than or equal to two, is

NPrim := {⟨x⟩ : x ≥ 2 and x is not a prime number}.

It is not obvious at all, whether or not NPrim is in the class P. In fact, it was shown only in 2002 that NPrim is in the class P.

Observation 6.3.5 The four languages above have the following in common: If someone gives us a “solution” for any given input, then we can easily, i.e., in polynomial time, verify whether or not this “solution” is a correct solution. Moreover, for any input to each of these four problems, there exists a “solution” whose length is polynomial in the length of the input.

Let us again consider the language kColor. Let G be a graph with vertex set V = {1, 2, …, m} and edge set E, and let k be a positive integer. We want to decide whether or not G is k-colorable. A “solution” is a coloring of the nodes using at most k different colors. That is, a solution is a sequence f_{1}, f_{2}, …, f_{m}. (Interpret this as: vertex i receives color f_{i}, 1 ≤ i ≤ m). This sequence is a correct solution if and only if

1. f_{i} ∈ {1, 2, …, k}, for all i with 1 ≤ i ≤ m, and

2. for all i with 1 ≤ i ≤ m, and for all j with 1 ≤ j ≤ m, if (i, j) ∈ E, then f_{i} ≠ f_{j}.

If someone gives us this solution (i.e., the sequence f_{1}, f_{2}, …, f_{m}), then we can verify in polynomial time whether or not these two conditions are satisfied. The length of this solution is O(m log k): for each i, we need about log k bits to represent f_{i}. Hence, the length of the solution is polynomial in the length of the input, i.e., it is polynomial in the number of bits needed to represent the graph G and the number k.

For the Hamilton cycle problem, a solution consists of a sequence v_{1}, v_{2}, …, v_{m} of vertices. This sequence is a correct solution if and only if

1. {v_{1}, v_{2}, …, v_{m}} = {1, 2, …, m} and

2. {(v_{1}, v_{2}), (v_{2}, v_{3}), …, (v_{m−1}, v_{m}), (v_{m}, v_{1})} ⊆ E.

These two conditions can be verified in polynomial time. Moreover, the length of the solution is polynomial in the length of the input graph.

Consider the sum of subset problem. A solution is a sequence c_{1}, c_{2}, …, c_{m}. It is a correct solution if and only if

1. c_{i} ∈ {0, 1}, for all i with 1 ≤ i ≤ m, and

2. ∑_{i=1}^{m} c_{i}a_{i} = b.

Hence, the set I ⊆ {1, 2, …, m} in the definition of SOS is the set of indices i for which c_{i} = 1. Again, these two conditions can be verified in polynomial time, and the length of the solution is polynomial in the length of the input.

Finally, let us consider the language NPrim. Let x ≥ 2 be an integer. The integers a and b form a “solution” for x if and only if 1. 2 ≤ a < x,

2. 2 ≤ b < x, and

3. x = ab.

Clearly, these three conditions can be verified in polynomial time. Moreover, the length of this solution, i.e., the total number of bits in the binary representations of a and b, is polynomial in the number of bits in the binary representation of x.

Languages having the property that the correctness of a proposed “solution” can be verified in polynomial time, form the class NP:

Definition 6.3.6 A language A belongs to the class NP, if there exist a polynomial p and a language B ∈ P, such that for every string w,

w ∈ A ⟺ ∃s : |s| ≤ p(|w|) and ⟨w, s⟩ ∈ B.

In words, a language A is in the class NP, if for every string w, w ∈ A if and only if the following two conditions are satisfied:

1. There is a “solution” s, whose length |s| is polynomial in the length of w (i.e., |s| ≤ p(|w|), where p is a polynomial).

2. In polynomial time, we can verify whether or not s is a correct “solution” for w (i.e., ⟨w, s⟩ ∈ B and B ∈ P).

Hence, the language B can be regarded to be the “verification language”:

B = {⟨w, s⟩ : s is a correct “solution” for w}.

We have given already informal proofs of the fact that the languages kColor, HC , SOS, and NPrim are all contained in the class NP. Below, we formally prove that NPrim ∈ NP. To prove this claim, we have to specify the polynomial p and the language B ∈ P. First, we observe that

NPrim = {⟨x⟩ : there exist a and b in ℕ such that

2 ≤ a < x, 2 ≤ b < x and x = ab }. (6.1)

We define the polynomial p by p(n) := n + 2, and the language B as

B := {⟨x, a, b⟩ : x ≥ 2, 2 ≤ a < x, 2 ≤ b < x and x = ab}.

It is obvious that B ∈ P: For any three positive integers x, a, and b, we can verify in polynomial time whether or not ⟨x, a, b⟩ ∈ B. In order to do this, we only have to verify whether or not x ≥ 2, 2 ≤ a < x, 2 ≤ b < x, and x = ab. If all these four conditions are satisfied, then ⟨x, a, b⟩ ∈ B. If at least one of them is not satisfied, then ⟨x, a, b⟩ ∉ B.

It remains to show that for all x ∈ ℕ:

⟨x⟩ ∈ NPrim ⟺ ∃a, b : |⟨a, b⟩| ≤ |⟨x⟩| + 2 and ⟨x, a, b⟩ ∈ B.  (6.2)

(Remember that |⟨x⟩| denotes the number of bits in the binary representation of x; |⟨a, b⟩| denotes the total number of bits of a and b, i.e., |⟨a, b⟩| = |⟨a⟩| + |⟨b⟩|.)

Let x ∈ NPrim. It follows from (6.1) that there exist a and b in ℕ, such that 2 ≤ a < x, 2 ≤ b < x, and x = ab. Since x = ab ≥ 2 · 2 = 4 ≥ 2, it follows that ⟨x, a, b⟩ ∈ B. Hence, it remains to show that

|⟨a, b⟩| ≤ |⟨x⟩| + 2.

The binary representation of x contains ⌊log x⌋+1 bits, i.e., |⟨x⟩| = ⌊log x⌋+1. We have

|⟨a, b⟩| = |⟨a⟩| + |⟨b⟩|
= (⌊log a⌋ + 1) + (⌊log b⌋ + 1)
≤ log a + log b + 2
= log ab + 2
= log x + 2
≤ ⌊log x⌋ + 3
= |⟨x⟩| + 2.

This proves one direction of (6.2).

To prove the other direction, we assume that there are positive integers a and b, such that |⟨a, b⟩| ≤ |⟨x⟩| + 2 and ⟨x, a, b⟩ ∈ B. Then it follows immediately from (6.1) and the definition of the language B, that x ∈ NPrim. Hence, we have proved the other direction of (6.2). This completes the proof of the claim that

NPrim ∈ NP.

6.3.1 P is contained in NP

Intuitively, it is clear that P ⊆ NP, because a language is

• in P, if for every string w, it is possible to compute the “solution” s in polynomial time,

• in NP, if for every string w and for any given “solution” s, it is possible to verify in polynomial time whether or not s is a correct solution for w (hence, we do not need to compute the solution s ourselves, we only have to verify it).

We give a formal proof of this:

Theorem 6.3.7 P ⊆ NP.

Proof. Let A ∈ P. We will prove that A ∈ NP. Define the polynomial p by p(n) := 0 for all n ∈ ℕ_{0}, and define

B := {⟨w, ϵ⟩ : w ∈ A}.

Since A ∈ P, the language B is also contained in P. It is easy to see that

w ∈ A ⟺ ∃s : |s| ≤ p(|w|) = 0 and ⟨w, s⟩ ∈ B.

This completes the proof.

6.3.2 Deciding NP-languages in exponential time

Let us look again at the definition of the class NP. Let A be a language in this class. Then there exist a polynomial p and a language B ∈ P, such that for all strings w,

w ∈ A ⟺ ∃s : |s| ≤ p(|w|) and ⟨w, s⟩ ∈ B.  (6.3)

How do we decide whether or not any given string w belongs to the language A? If we can find a string s that satisfies the right-hand side in (6.3), then we know that w ∈ A. On the other hand, if there is no such string s, then w ∉ A. How much time do we need to decide whether or not such a string s exists?

Algorithm NonPrime

(∗ decides whether or not ⟨x⟩ ∈ NPrim ∗)

if x = 0 or x = 1 or x = 2 then return NO and terminate else a := 2;

while a < x

do if x mod a = 0 then return YES and terminate else a := a + 1 endif endwhile; return NO

endif

Figure 6.3: An algorithm that decides whether or not a number x is contained in the language NPrim.

For example, let A be the language NPrim, and let x ∈ ℕ. The algorithm in Figure 6.3 decides whether or not ⟨x⟩ ∈ NPrim.

It is clear that this algorithm is correct. Let n be the length of the binary representation of x, i.e., n = ⌊log x⌋ + 1. If x > 2 and x is a prime number, then the while-loop makes x−2 iterations. Therefore, since n−1 = ⌊log x⌋ ≤ log x, the running time of this algorithm is at least

x − 2 ≥ 2^{n−1} − 2,

i.e., it is at least exponential in the length of the input.

We now prove that every language in NP can be decided in exponential time. Let A be an arbitrary language in NP. Let p be the polynomial, and let B ∈ P be the language such that for all strings w,

w ∈ A ⟺ ∃s : |s| ≤ p(|w|) and ⟨w, s⟩ ∈ B.  (6.4)

The following algorithm decides, for any given string w, whether or not w ∈ A. It does so by looking at all possible strings s for which |s| ≤ p(|w|):

for all s with |s| ≤ p(|w|)
do if ⟨w, s⟩ ∈ B

then return YES and terminate endif endfor; return NO

The correctness of the algorithm follows from (6.4). What is the running time? We assume that w and s are represented as binary strings. Let n be the length of the input, i.e., n = |w|.

How many binary strings s are there whose length is at most p(|w|)? Any such s can be described by a sequence of length p(|w|) = p(n), consisting of the symbols “0”, “1”, and the blank symbol. Hence, there are at most 3^{p(n)} many binary strings s with |s| ≤ p(n). Therefore, the for-loop makes at most 3^{p(n)} iterations.

Since B ∈ P, there is an algorithm and a polynomial q, such that this algorithm, when given any input string z, decides in q(|z|) time, whether or not z ∈ B. This input z has the form ⟨w, s⟩, and we have

|z| = |w| + |s| ≤ |w| + p(|w|) = n + p(n).

It follows that the total running time of our algorithm that decides whether or not w ∈ A, is bounded from above by

3^{p(n)} · q(n + p(n)) ≤ 2^{2p(n)} · q(n + p(n))
≤ 2^{2p(n)} · 2^{q(n+p(n))}
= 2^{p′(n)},

where p′ is the polynomial that is defined by p′(n) := 2p(n) + q(n + p(n)). If we define the class EXP as

EXP := {A : there exists a polynomial p, such that A can be
decided in time 2^{p(n)} } ,

then we have proved the following theorem.

Theorem 6.3.8 NP ⊆ EXP.

6.3.3 Summary

• P ⊆ NP. It is not known whether P is a proper subclass of NP, or whether P = NP. This is one of the most important open problems in computer science. If you can solve this problem, then you will get one million dollars; not from us, but from the Clay Mathematics Institute, see

      http://www.claymath.org/prizeproblems/index.htm

Most people believe that P is a proper subclass of NP.

• NP ⊆ EXP, i.e., each language in NP can be decided in exponential time. It is not known whether NP is a proper subclass of EXP, or whether NP = EXP.

• It follows from P ⊆ NP and NP ⊆ EXP, that P ⊆ EXP. It can be shown that P is a proper subset of EXP, i.e., there exist languages that can be decided in exponential time, but that cannot be decided in polynomial time.

• P is the class of those languages that can be decided efficiently, i.e., in polynomial time. Sets that are not in P, are not efficiently decidable.

6.4 Non-deterministic algorithms

The abbreviation NP stands for Non-deterministic Polynomial time. The algorithms that we have considered so far are deterministic, which means that at any time during the computation, the next computation step is uniquely determined. In a non-deterministic algorithm, there are one or more possibilities for being the next computation step, and the algorithm chooses one of them.

To give an example, we consider the language SOS, see Example 6.3.3. Let m, a_{1}, a_{2}, …, a_{m}, and b be elements of ℕ_{0}. Then

⟨a_{1}, a_{2}, …, a_{m}, b⟩ ∈ SOS ⟺ there exist c_{1}, c_{2}, …, c_{m} ∈ {0, 1},
such that ∑_{i=1}^{m} c_{i}a_{i} = b.

The following non-deterministic algorithm decides the language SOS:

Algorithm SOS(m, a_{1}, a_{2}, …, a_{m}, b): s := 0; for i := 1 to m do s := s | s := s + a_{i} endfor; if s = b then return YES else return NO endif

The line

s := s | s := s + a_{i}

means that either the instruction “s := s” or the instruction “s := s + a_{i}” is executed.

Let us assume that ⟨a_{1}, a_{2}, …, a_{m}, b⟩ ∈ SOS. Then there are c_{1}, c_{2}, …, c_{m} ∈ {0, 1} such that ∑_{i=1}^{m} c_{i}a_{i} = b. Assume our algorithm does the following, for each i with 1 ≤ i ≤ m: In the i-th iteration,

• if c_{i} = 0, then it executes the instruction “s := s”,

• if c_{i} = 1, then it executes the instruction “s := s + a_{i}”.

Then after the for-loop, we have s = b, and the algorithm returns YES; hence, the algorithm has correctly found out that ⟨a_{1}, a_{2}, …, a_{m}, b⟩ ∈ SOS. In other words, in this case, there exists at least one accepting computation.

On the other hand, if ⟨a_{1}, a_{2}, …, a_{m}, b⟩ ∉ SOS, then the algorithm always returns NO, no matter which of the two instructions is executed in each iteration of the for-loop. In this case, there is no accepting computation.

Definition 6.4.1 Let M be a non-deterministic algorithm. We say that M accepts a string w, if there exists at least one computation that, on input w, returns YES.

Definition 6.4.2 We say that a non-deterministic algorithm M decides a language A in time T, if for every string w, the following holds: w ∈ A if and only if there exists at least one computation that, on input w, returns YES and that takes at most T (|w|) time.

The non-deterministic algorithm that we have seen above decides the language SOS in linear time: Let ⟨a_{1}, a_{2}, …, a_{m}, b⟩ ∈ SOS, and let n be the length of this input. Then

n = |⟨a_{1}⟩| + |⟨a_{2}⟩| + … + |⟨a_{m}⟩| + |⟨b⟩| ≥ m.

For this input, there is a computation that returns YES and that takes O(m) = O(n) time.

As in Section 6.2, we define the notion of “polynomial time” for nondeterministic algorithms. The following theorem relates this notion to the class NP that we defined in Definition 6.3.6.

Theorem 6.4.3 A language A is in the class NP if and only if there exists a non-deterministic Turing machine (or Java program) that decides A in polynomial time.

6.5 NP-complete languages

Languages in the class P are considered easy, i.e., they can be decided in polynomial time. People believe (but cannot prove) that P is a proper subclass of NP. If this is true, then there are languages in NP that are hard, i.e., cannot be decided in polynomial time.

Intuition tells us that if P ≠ NP, then the hardest languages in NP are not contained in P. These languages are called NP-complete. In this section, we will give a formal definition of this concept.

If we want to talk about the “hardest” languages in NP, then we have to be able to compare two languages according to their “difficulty”. The idea is as follows: We say that a language B is “at least as hard” as a language A, if the following holds: If B can be decided in polynomial time, then A can also be decided in polynomial time.

Definition 6.5.1 Let A ⊆ {0, 1}^{∗} and B ⊆ {0, 1}^{∗} be languages. We say that A ≤_{P} B, if there exists a function

f : {0, 1}^{∗} → {0, 1}^{∗}

such that

1. f ∈ FP and

2. for all strings w in {0, 1}^{∗},

w ∈ A ⟺ f(w) ∈ B.

If A ≤_{P} B, then we also say that “B is at least as hard as A”, or “A is polynomial-time reducible to B”.

We first show that this formal definition is in accordance with the intuitive definition given above.

Theorem 6.5.2 Let A and B be languages such that B ∈ P and A ≤_{P} B. Then A ∈ P.

Proof. Let f : {0, 1}^{∗} → {0, 1}^{∗} be the function in FP for which

w ∈ A ⟺ f(w) ∈ B.  (6.5)

The following algorithm decides whether or not any given binary string w is in A:

u := f(w); if u ∈ B then return YES else return NO endif

The correctness of this algorithm follows immediately from (6.5). So it remains to show that the running time is polynomial in the length of the input string w.

Since f ∈ FP, there exists a polynomial p such that the function f can be computed in time p. Similarly, since B ∈ P, there exists a polynomial q, such that the language B can be decided in time q.

Let n be the length of the input string w, i.e., n = |w|. Then the length of the string u is less than or equal to p(|w|) = p(n). (Why?) Therefore, the running time of our algorithm is bounded from above by

p(|w|) + q(|u|) ≤ p(n) + q(p(n)).

Since the function p′, defined by p′(n) := p(n)+q(p(n)), is a polynomial, this proves that A ∈ P.

The following theorem states that the relation ≤_{P} is reflexive and transitive. We leave the proof as an exercise.

Theorem 6.5.3 Let A, B, and C be languages. Then 1. A ≤_{P} A, and

2. if A ≤_{P} B and B ≤_{P} C, then A ≤_{P} C.

We next show that the languages in P are the easiest languages in NP:

Theorem 6.5.4 Let A be a language in P, and let B be an arbitrary language such that B ≠ ∅ and B ≠ {0, 1}^{∗}. Then A ≤_{P} B.

Proof. We choose two strings u and v in {0, 1}^{∗}, such that u ∈ B and v ∉ B. (Observe that this is possible.) Define the function f : {0, 1}^{∗} → {0, 1}^{∗} by

f(w) := \{ \begin{matrix}u & if w ∈ A, \\ v & if w ∉ A. \end{matrix}

Then it is clear that for any binary string w,

w ∈ A ⟺ f(w) ∈ B.

Since A ∈ P, the function f can be computed in polynomial time, i.e., f ∈ FP.

6.5.1 Two examples of reductions

Sum of subsets and knapsacks

We start with a simple reduction. Consider the two languages

SOS := {⟨a_{1}, …, a_{m}, b⟩ : m, a_{1}, …, a_{m}, b ∈ ℕ_{0} and there exist
c_{1}, …, c_{m} ∈ {0, 1}, such that ∑_{i=1}^{m} c_{i}a_{i} = b}

and

KS := {⟨w_{1}, …, w_{m}, k_{1}, …, k_{m}, W, K⟩ :
m, w_{1}, …, w_{m}, k_{1}, …, k_{m}, W, K ∈ ℕ_{0}
and there exist c_{1}, …, c_{m} ∈ {0, 1},
such that ∑_{i=1}^{m} c_{i}w_{i} ≤ W and ∑_{i=1}^{m} c_{i}k_{i} ≥ K}.

The notation KS stands for knapsack: We have m pieces of food. The i-th piece has weight w_{i} and contains k_{i} calories. We want to decide whether or not we can fill our knapsack with a subset of the pieces of food such that the total weight is at most W, and the total amount of calories is at least K.

Theorem 6.5.5 SOS ≤_{P} KS.

Proof. Let us first see what we have to show. According to Definition 6.5.1, we need a function f ∈ FP, that maps input strings for SOS to input strings for KS, in such a way that

⟨a_{1}, …, a_{m}, b⟩ ∈ SOS ⟺ f(⟨a_{1}, …, a_{m}, b⟩) ∈ KS.

In order for f(⟨a_{1}, …, a_{m}, b⟩) to be an input string for KS, this function value has to be of the form

f(⟨a_{1}, …, a_{m}, b⟩) = ⟨w_{1}, …, w_{m}, k_{1}, …, k_{m}, W, K⟩.

We define

f(⟨a_{1}, …, a_{m}, b⟩) := ⟨a_{1}, …, a_{m}, a_{1}, …, a_{m}, b, b⟩.

It is clear that f ∈ FP. We have

⟨a_{1}, …, a_{m}, b⟩ ∈ SOS

⟺ there exist c_{1}, …, c_{m} ∈ {0, 1} such that ∑_{i=1}^{m} c_{i}a_{i} = b
⟺ there exist c_{1}, …, c_{m} ∈ {0, 1} such that ∑_{i=1}^{m} c_{i}a_{i} ≤ b and ∑_{i=1}^{m} c_{i}a_{i} ≥ b
⟺ ⟨a_{1}, …, a_{m}, a_{1}, …, a_{m}, b, b⟩ ∈ KS
⟺ f(⟨a_{1}, …, a_{m}, b⟩) ∈ KS.

Cliques and Boolean formulas

We will define two languages A = 3SAT and B = Clique that have, at first sight, nothing to do with each other. Then we show that, nevertheless, A ≤_{P} B.

Let G be a graph with vertex set V and edge set E. A subset V′ of V is called a clique, if each pair of distinct vertices in V′ is connected by an edge in E. We define the following language:

Clique := {⟨G, k⟩ : k ∈ ℕ and G has a clique with k vertices}.

We encourage you to prove the following claim:

Theorem 6.5.6 Clique ∈ NP.

Next we consider Boolean formulas φ, with variables x_{1}, x_{2}, …, x_{m}, having the form

φ = C_{1} ∧ C_{2} ∧ … ∧ C_{k},  (6.6)

where each C_{i}, 1 ≤ i ≤ k, is of the form

C_{i} = ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ ℓ_{3}^{i}.

Each ℓ_{a}^{i} is either a variable or the negation of a variable. An example of such a formula is

φ = (x_{1} ∨ ¬x_{1} ∨ ¬x_{2}) ∧ (x_{3} ∨ x_{2} ∨ x_{4}) ∧ (¬x_{1} ∨ ¬x_{3} ∨ ¬x_{4}).

A formula φ of the form (6.6) is said to be satisfiable, if there exists a truthvalue in {0, 1} for each of the variables x_{1}, x_{2}, …, x_{m}, such that the entire formula φ is true. Our example formula is satisfiable: If we take x_{1} = 0 and x_{2} = 1, and give x_{3} and x_{4} an arbitrary value, then

φ = (0 ∨ 1 ∨ 0) ∧ (x_{3} ∨ 1 ∨ x_{4}) ∧ (1 ∨ ¬x_{3} ∨ ¬x_{4}) = 1.

We define the following language:

3SAT := {⟨φ⟩ : φ is of the form (6.6) and is satisfiable}.

Again, we encourage you to prove the following claim:

Theorem 6.5.7 3SAT ∈ NP.

Observe that the elements of Clique (which are pairs consisting of a graph and a positive integer) are completely different from the elements of 3SAT (which are Boolean formulas). We will show that 3SAT ≤_{P} Clique. Recall that this means the following: If the language Clique can be decided in polynomial time, then the language 3SAT can also be decided in polynomial time. In other words, any polynomial-time algorithm that decides Clique can be converted to a polynomial-time algorithm that decides 3SAT .

Theorem 6.5.8 3SAT ≤_{P} Clique.

Proof. We have to show that there exists a function f ∈ FP, that maps input strings for 3SAT to input strings for Clique, such that for each Boolean formula φ that is of the form (6.6),

⟨φ⟩ ∈ 3SAT ⟺ f(⟨φ⟩) ∈ Clique.

The function f maps the binary string encoding an arbitrary Boolean formula φ to a binary string encoding a pair (G, k), where G is a graph and k is a positive integer. We have to define this function f in such a way that

φ is satisfiable ⟺ G has a clique with k vertices.

Let

φ = C_{1} ∧ C_{2} ∧ … ∧ C_{k}

be an arbitrary Boolean formula in the variables x_{1}, x_{2}, …, x_{m}, where each C_{i}, 1 ≤ i ≤ k, is of the form

C_{i} = ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ ℓ_{3}^{i}.

Remember that each ℓ_{a}^{i} is either a variable or the negation of a variable.

The formula φ is mapped to the pair (G, k), where the vertex set V and the edge set E of the graph G are defined as follows:

• V = {v_{1}^{1}, v_{2}^{1}, v_{3}^{1}, …, v_{1}^{k}, v_{2}^{k}, v_{3}^{k}}. The idea is that each vertex v_{a}^{i} corresponds to one term ℓ_{a}^{i}.

• The pair (v_{a}^{i}, v_{b}^{j}) of vertices form an edge in E if and only if

– i ≠ j and

– ℓ_{a}^{i} is not the negation of ℓ_{b}^{j}.

To give an example, let φ be the Boolean formula

φ = (x_{1} ∨ ¬x_{2} ∨ ¬x_{3}) ∧ (¬x_{1} ∨ x_{2} ∨ x_{3}) ∧ (x_{1} ∨ x_{2} ∨ x_{3}),  (6.7)

i.e., k = 3, C_{1} = x_{1} ∨ ¬x_{2} ∨ ¬x_{3}, C_{2} = ¬x_{1} ∨ x_{2} ∨ x_{3}, and C_{3} = x_{1} ∨ x_{2} ∨ x_{3}. The graph G that corresponds to this formula is given in Figure 6.4.

It is not difficult to see that the function f can be computed in polynomial time. So it remains to prove that

φ is satisfiable ⟺ G has a clique with k vertices.  (6.8)

¬x_{2}  ¬x_{3} x_{1} ¬x_{1} x_{2} x_{3}  x_{1} x_{2} x_{3}

Figure 6.4: The formula φ in (6.7) is mapped to this graph. The vertices on the top represent C_{1}; the vertices on the left represent C_{2}; the vertices on the right represent C_{3}.

To prove this, we first assume that the formula

φ = C_{1} ∧ C_{2} ∧ … ∧ C_{k}

is satisfiable. Then there exists a truth-value in {0, 1} for each of the variables x_{1}, x_{2}, …, x_{m}, such that the entire formula φ is true. Hence, for each i with 1 ≤ i ≤ k, there is at least one term ℓ_{a}^{i} in

C_{i} = ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ ℓ_{3}^{i}

that is true (i.e., has value 1).

Let V′ be the set of vertices obtained by choosing for each i, 1 ≤ i ≤ k, exactly one vertex v_{a}^{i} such that ℓ_{a}^{i} has value 1.

It is clear that V′ contains exactly k vertices. We claim that this set is a clique in G. To prove this claim, let v_{a}^{i} and v_{b}^{j} be two distinct vertices in V′. It follows from the definition of V′ that i ≠ j and ℓ_{a}^{i} = ℓ_{b}^{j} = 1. Hence, ℓ_{a}^{i} is not the negation of ℓ_{b}^{j}. But this means that the vertices v_{a}^{i} and v_{b}^{j} are connected by an edge in G.

This proves one direction of (6.8). To prove the other direction, we assume that the graph G contains a clique V′ with k vertices.

The vertices of G consist of k groups, where each group contains exactly three vertices. Since vertices within the same group are not connected by edges, the clique V′ contains exactly one vertex from each group. Hence, for each i with 1 ≤ i ≤ k, there is exactly one a, such that v_{a}^{i} ∈ V′. Consider the corresponding term ℓ_{a}^{i}. We know that this term is either a variable or the negation of a variable, i.e., ℓ_{a}^{i} is either of the form x_{j} or of the form ¬x_{j}. If ℓ_{a}^{i} = x_{j}, then we give x_{j} the truth-value 1. Otherwise, we have ℓ_{a}^{i} = ¬x_{j}, in which case we give x_{j} the truth-value 0. Since V′ is a clique, each variable gets at most one truth-value. If a variable has no truth-value yet, then we give it an arbitrary truth-value.

If we substitute these truth-values into φ, then the entire formula has value 1. Hence, φ is satisfiable.

In order to get a better understanding of this proof, you should verify the proof for the formula φ in (6.7) and the graph G in Figure 6.4.

6.5.2 Definition of NP-completeness

Reductions, as defined in Definition 6.5.1, allow us to compare two language according to their difficulty. A language B in NP is called NP-complete, if B belongs to the most difficult languages in NP; in other words, B is at least as hard as any other language in NP.

Definition 6.5.9 Let B ⊆ {0, 1}^{∗} be a language. We say that B is NPcomplete, if

1. B ∈ NP and

2. A ≤_{P} B, for every language A in NP.

Theorem 6.5.10 Let B be an NP-complete language. Then

B ∈ P ⟺ P = NP.

Proof. Intuitively, this theorem should be true: If the language B is in P, then B is an easy language. On the other hand, since B is NP-complete, it belongs to the most difficult languages in NP. Hence, the most difficult language in NP is easy. But then all languages in NP must be easy, i.e., P = NP.

We give a formal proof. Let us first assume that B ∈ P. We already know that P ⊆ NP. Hence, it remains to show that NP ⊆ P. Let A be an arbitrary language in NP. Since B is NP-complete, we have A ≤_{P} B. Then, by Theorem 6.5.2, we have A ∈ P.

To prove the converse, assume that P = NP. Since B ∈ NP, it follows immediately that B ∈ P.

Theorem 6.5.11 Let B and C be languages, such that C ∈ NP and B ≤_{P} C. If B is NP-complete, then C is also NP-complete.

Proof. First, we give an intuitive explanation of the claim: By assumption, B belongs to the most difficult languages in NP, and C is at least as hard as B. Since C ∈ NP, it follows that C belongs to the most difficult languages in NP. Hence, C is NP-complete.

To give a formal proof, we have to show that A ≤_{P} C, for all languages A in NP. Let A be an arbitrary language in NP. Since B is NP-complete, we have A ≤_{P} B. Since B ≤_{P} C, it follows from Theorem 6.5.3, that A ≤_{P} C. Therefore, C is NP-complete.

Theorem 6.5.11 can be used to prove the NP-completeness of languages: Let C be a language, and assume that we want to prove that C is NPcomplete. We can do this in the following way:

1. We first prove that C ∈ NP.

2. Then we find a language B that looks “similar” to C, and for which we already know that it is NP-complete.

3. Finally, we prove that B ≤_{P} C.

4. Then, Theorem 6.5.11 tells us that C is NP-complete.

Of course, this leads to the question “How do we know that the language B is NP-complete?” In order to apply Theorem 6.5.11, we need a “first” NPcomplete language; the NP-completeness of this language must be proven using Definition 6.5.9.

Observe that it is not clear at all that there exist NP-complete languages! For example, consider the language 3SAT . If we want to use Definition 6.5.9 to show that this language is NP-complete, then we have to show that • 3SAT ∈ NP. We know from Theorem 6.5.7 that this is true.

• A ≤_{P} 3SAT , for every language A ∈ NP. Hence, we have to show this for languages A such as kColor, HC , SOS, NPrim, KS, Clique, and for infinitely many other languages.

In 1973, Cook has exactly done this: He showed that the language 3SAT is NP-complete. Since his proof is rather technical, we will prove the NPcompleteness of another language.

6.5.3 An NP-complete domino game

We are given a finite collection of tile types. For each such type, there are arbitrarily many tiles of this type. A tile is a square that is partitioned into four triangles. Each of these triangles contains a symbol that belongs to a finite alphabet Σ. Hence, a tile looks as follows:

[figure: a; b; c; d]

We are also given a square frame, consisting of cells. Each cell has the same size as a tile, and contains a symbol of Σ.

The problem is to decide whether or not this domino game has a solution. That is, can we completely fill the frame with tiles such that

• for any two neighboring tiles s and s′, the two triangles of s and s′ that touch each other contain the same symbol, and

• each triangle that touches the frame contains the same symbol as the cell of the frame that is touched by this triangle.

There is one final restriction: The orientation of the tiles is fixed, they cannot be rotated.

Let us give a formal definition of this problem. We assume that the symbols belong to the finite alphabet Σ = {0, 1}^{m}, i.e., each symbol is encoded as a bit-string of length m. Then, a tile type can be encoded as a tuple of four bit-strings, i.e., as an element of Σ^{4}. A frame consisting of t rows and t columns can be encoded as a string in Σ^{4t}.

We denote the language of all solvable domino games by Domino:

Domino := {⟨m, k, t, R, T_{1}, …, T_{k}⟩ :

m ≥ 1, k ≥ 1, t ≥ 1, R ∈ Σ^{4t}, T_{i} ∈ Σ^{4}, 1 ≤ i ≤ k, frame R can be filled using tiles of types T_{1}, …, T_{k}.}

We will prove the following theorem.

Theorem 6.5.12 The language Domino is NP-complete.

Proof. It is clear that Domino ∈ NP: A solution consists of a t × t matrix, in which the (i, j)-entry indicates the type of the tile that occupies position (i, j) in the frame. The number of bits needed to specify such a solution is polynomial in the length of the input. Moreover, we can verify in polynomial time whether or not any given “solution” is correct.

It remains to show that

A ≤_{P} Domino, for every language A in NP.

Let A be an arbitrary language in NP. Then there exist a polynomial p and a non-deterministic Turing machine M, that decides the language A in time p. We may assume that this Turing machine has only one tape.

On input w = a_{1}a_{2} … a_{n}, the Turing machine M starts in the start state z_{0}, with its tape head on the cell containing the symbol a_{1}. We may assume that during the entire computation, the tape head never moves to the left of this initial cell. Hence, the entire computation “takes place” in and to the right of the initial cell. We know that

w ∈ A ⟺ on input w, there exists an accepting computation

that makes at most p(n) computation steps.

At the end of such an accepting computation, the tape only contains the symbol 1, which we may assume to be in the initial cell, and M is in the final state z_{1}. In this case, we may assume that the accepting computation makes exactly p(n) computation steps. (If this is not the case, then we extend the computation using the instruction z_{1}1 → z_{1}1N.)

We need one more technical detail: We may assume that za → z′bR and za′ → z′′b′L are not both instructions of M. Hence, the state of the Turing machine uniquely determines the direction in which the tape head moves.

We have to define a domino game, that depends on the input string w and the Turing machine M, such that

w ∈ A ⟺ this domino game is solvable.

The idea is to encode an accepting computation of the Turing machine M as a solution of the domino game. In order to do this, we use a frame in which each row corresponds to one computation step. This frame consists of p(n) rows. Since an accepting computation makes exactly p(n) computation steps, and since the tape head never moves to the left of the initial cell, this tape head can visit only p(n) cells. Therefore, our frame will have p(n) columns.

The domino game will use the following tile types:

1. For each symbol a in the alphabet of the Turing machine M:

[figure: #; a; #; a]

Intuition: Before and after the computation step, the tape head is not on this cell.

2. For each instruction za → z′bR of the Turing machine M:

[figure: #; (z, a); z′; b]

Intuition: Before the computation step, the tape head is on this cell; the tape head makes one step to the right.

3. For each instruction za → z′bL of the Turing machine M:

[figure: z′; (z, a); #; b]

Intuition: Before the computation step, the tape head is on this cell; the tape head makes one step to the left.

4. For each instruction za → z′bN of the Turing machine M:

[figure: #; (z, a); #; (z′, b)]

Intuition: Before and after the computation step, the tape head is on this cell.

5. For each state z and for each symbol a in the alphabet of the Turing machine M, there are two tile types:

[figure: z; a; #; (z, a); #; a; z; (z, a)]

Intuition: The leftmost tile indicates that the tape head enters this cell from the left; the righmost tile indicates that the tape head enters this cell from the right.

This specifies all tile types. The p(n) × p(n) frame is given in Figure 6.5. The top row corresponds to the start of the computation, whereas the bottom row corresponds to the end of the computation. The left and right columns correspond to the part of the tape in which the tape head can move.

The encodings of these tile types and the frame can be computed in polynomial time.

It can be shown that, for any input string w, any accepting computation of length p(n) of the Turing machine M can be encoded as a solution of this domino game. Conversely, any solution of this domino game can be “translated” to an accepting computation of length p(n) of M, on input string w. Hence, the following holds.

w ∈ A ⟺ there exists an accepting computation that makes

p(n) computation steps

⟺ the domino game is solvable.

(z_{0}, a_{1}) a_{2}  …  a_{n}  □  …  □ # # #  # # ⋮ # ⋮  □  □  □ □ □ … (z_{1}, 1)  p(n)  p(n)

Figure 6.5: The p(n) × p(n) frame of the domino game.

Therefore, we have A ≤_{P} Domino. Hence, the language Domino is NPcomplete.

An example of a domino game

We have defined the domino game corresponding to a Turing machine that solves a decision problem. Of course, we can also do this for Turing machines that compute functions. In this section, we will exactly do this for a Turing machine that computes the successor function x → x + 1.

We will design a Turing machine with one tape, that gets as input the binary representation of a natural number x, and that computes the binary representation of x + 1.

Start of the computation: The tape contains a 0 followed by the binary representation of the integer x ∈ ℕ_{0}. The tape head is on the leftmost bit (which is 0), and the Turing machine is in the start state z_{0}. Here is an example, where x = 431:

0 1 1 0 1 0 1 1 1 1 □^{}

End of the computation: The tape contains the binary representation of the number x + 1. The tape head is on the rightmost 1, and the Turing machine is in the final state z_{1}. For our example, the tape looks as follows:

0 1 1 0 1 1 0 0 0 0 □^{}

Our Turing machine will use the following states:

z_{0} : start state; tape head moves to the right
z_{1} : final state
z_{2} : tape head moves to the left; on its way to the left, it has not read 0

The Turing machine has the following instructions:

z_{0}0 → z_{0}0R  z_{2}1 → z_{2}0L
z_{0}1 → z_{0}1R  z_{2}0 → z_{1}1N
z_{0}□ → z_{2}□L

In Figure 6.6, you can see the sequence of states and tape contents of this Turing machine on input x = 11.

We now construct the domino game that corresponds to the computation of this Turing machine on input x = 11. Following the general construction in Section 6.5.3, we obtain the following tile types:

1. The three symbols of the alphabet yield three tile types:

[figure: #; #; 0; 0; #; #; 1; 1; #; #; □; □]

\begin{matrix}(z_{0}, 0) & 1 & 0 & 1 & 1 & □ \\ 0 & (z_{0}, 1) & 0 & 1 & 1 & □ \\ 0 & 1 & (z_{0}, 0) & 1 & 1 & □ \\ 0 & 1 & 0 & (z_{0}, 1) & 1 & □ \\ 0 & 1 & 0 & 1 & (z_{0}, 1) & □ \\ 0 & 1 & 0 & 1 & 1 & (z_{0}, □) \\ 0 & 1 & 0 & 1 & (z_{2}, 1) & □ \\ 0 & 1 & 0 & (z_{2}, 1) & 0 & □ \\ 0 & 1 & (z_{2}, 0) & 0 & 0 & □ \\ 0 & 1 & (z_{1}, 1) & 0 & 0 & □ \end{matrix}

Figure 6.6: The computation of the Turing machine on input x = 11. The pair (state,symbol) indicates the position of the tape head.

2. The five instructions of the Turing machine yield five tile types:

[figure: #; z_{0}; 0; (z_{0}, 0); #; z_{0}; 1; (z_{0}, 1); z_{2}; #; □; (z_{0}, □); z_{2}; #; 0; (z_{2}, 1); #; #; (z_{1}, 1); (z_{2}, 0)]

3. The states z_{0} and z_{2}, and the three symbols of the alphabet yield twelve tile types:

[figure: #; z_{0}; (z_{0}, 0); 0; z_{0}; #; (z_{0}, 0); 0; #; z_{0}; (z_{0}, 1); 1; z_{0}; #; (z_{0}, 1); 1; #; z_{0}; (z_{0}, □); □; z_{0}; #; (z_{0}, □); □; #; z_{2}; (z_{2}, 0); 0; z_{2}; #; (z_{2}, 0); 0; #; z_{2}; (z_{2}, 1); 1; z_{2}; #; (z_{2}, 1); 1; #; z_{2}; (z_{2}, □); □; z_{2}; #; (z_{2}, □); □]

The computation of the Turing machine on input x = 11 consists of nine computation steps. During this computation, the tape head visits exactly six cells. Therefore, the frame of the domino game has nine rows and six columns. This frame is given in Figure 6.7. In Figure 6.8, you find the solution of the domino game. Observe that this solution is nothing but an equivalent way of writing the computation of Figure 6.6. Hence, the computation of the Turing machine corresponds to a solution of the domino game; in fact, the converse also holds.

6.5.4 Examples of NP-complete languages

In Section 6.5.3, we have shown that Domino is NP-complete. Using this result, we will apply Theorem 6.5.11 to prove the NP-completeness of some other languages.

Satisfiability

We consider Boolean formulas φ, in the variables x_{1}, x_{2}, …, x_{m}, having the form

φ = C_{1} ∧ C_{2} ∧ … ∧ C_{k},  (6.9)

where each C_{i}, 1 ≤ i ≤ k, is of the following form:

C_{i} = ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ … ∨ ℓ_{k_{i}}^{i}.

Each ℓ_{j}^{i} is either a variable or the negation of a variable. Such a formula φ is said to be satisfiable, if there exists a truth-value in {0, 1} for each of the variables x_{1}, x_{2}, …, x_{m}, such that the entire formula φ is true. We define the following language:

SAT := {⟨φ⟩ : φ is of the form (6.9) and is satisfiable}.

We will prove that SAT is NP-complete.

It is clear that SAT ∈ NP. If we can show that

Domino ≤_{P} SAT ,

then it follows from Theorem 6.5.11 that SAT is NP-complete. (In Theorem 6.5.11, take B := Domino and C := SAT .)

Hence, we need a function f ∈ FP, that maps input strings for Domino to input strings for SAT , in such a way that for every domino game D, the following holds:

domino game D is solvable ⟺ the formula encoded by the string f(⟨D⟩) is satisfiable.  (6.10)

Let us consider an arbitrary domino game D. Let k be the number of tile types, and let the frame have t rows and t columns. We denote the tile types by T_{1}, T_{2}, …, T_{k}.

We map this domino game D to a Boolean formula φ, such that (6.10) holds. The formula φ will have variables

x_{ijℓ}, 1 ≤ i ≤ t, 1 ≤ j ≤ t, 1 ≤ ℓ ≤ k.

These variables can be interpretated as follows:

x_{ijℓ} = 1 ⟺ there is a tile of type T_{ℓ} at position (i, j) of the frame.

We define:

• For all i and j with 1 ≤ i ≤ t and 1 ≤ j ≤ t:

C_{ij}^{1} := x_{ij1} ∨ x_{ij2} ∨ … ∨ x_{ijk}.

This formula expresses the condition that there is at least one tile at position (i, j).

• For all i, j, ℓ and ℓ′ with 1 ≤ i ≤ t, 1 ≤ j ≤ t, and 1 ≤ ℓ < ℓ′ ≤ k:

C_{ijℓℓ′}^{2} := ¬x_{ijℓ} ∨ ¬x_{ijℓ′}.

This formula expresses the condition that there is at most one tile at position (i, j).

• For all i, j, ℓ and ℓ′ with 1 ≤ i ≤ t, 1 ≤ j < t, 1 ≤ ℓ ≤ k and 1 ≤ ℓ′ ≤ k, such that i < t and the right symbol on a tile of type T_{ℓ} is not equal to the left symbol on a tile of type T_{ℓ′}:

C_{ijℓℓ′}^{3} := ¬x_{ijℓ} ∨ ¬x_{i,j+1,ℓ′}.

This formula expresses the condition that neighboring tiles in the same row “fit” together. There are symmetric formulas for neighboring tiles in the same column.

• For all j and ℓ with 1 ≤ j ≤ t and 1 ≤ ℓ ≤ k, such that the top symbol on a tile of type T_{ℓ} is not equal to the symbol at position j of the upper boundary of the frame:

C_{jℓ}^{4} := ¬x_{1jℓ}.

This formula expresses the condition that tiles that touch the upper boundary of the frame “fit” there. There are symmetric formulas for the lower, left, and right boundaries of the frame.

The formula φ is the conjunction of all these formulas C_{ij}^{1}, C_{ijℓℓ′}^{2}, C_{ijℓℓ′}^{3}, and C_{jℓ}^{4}. The complete formula φ consists of

O(t^{2}k + t^{2}k^{2} + t^{2}k^{2} + tk) = O(t^{2}k^{2})

terms, i.e., its length is polynomial in the length of the domino game. This implies that φ can be constructed in polynomial time. Hence, the function f that maps the domino game D to the Boolean formula φ, is in the class FP. It is not difficult to see that (6.10) holds for this function f. Therefore, we have proved the following result.

Theorem 6.5.13 The language SAT is NP-complete.

In Section 6.5.1, we have defined the language 3SAT .

Theorem 6.5.14 The language 3SAT is NP-complete.

Proof. It is clear that 3SAT ∈ NP. If we can show that

SAT ≤_{P} 3SAT ,

then the claim follows from Theorem 6.5.11. Let

φ = C_{1} ∧ C_{2} ∧ … ∧ C_{k}

be an input for SAT , in the variables x_{1}, x_{2}, …, x_{m}. We map φ, in polynomial time, to an input φ′ for 3SAT , such that

φ is satisfiable ⟺ φ′ is satisfiable.  (6.11)

For each i with 1 ≤ i ≤ k, we do the following. Consider

C_{i} = ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ … ∨ ℓ_{k_{i}}^{i}.

• If k_{i} = 1, then we define

C′_{i} := ℓ_{1}^{i} ∨ ℓ_{1}^{i} ∨ ℓ_{1}^{i}.

• If k_{i} = 2, then we define

C′_{i} := ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ ℓ_{2}^{i}.

• If k_{i} = 3, then we define

C′_{i} := C_{i}.

• If k_{i} ≥ 4, then we define

C′_{i} := (ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ z_{1}^{i}) ∧ (¬z_{1}^{i} ∨ ℓ_{3}^{i} ∨ z_{2}^{i}) ∧ (¬z_{2}^{i} ∨ ℓ_{4}^{i} ∨ z_{3}^{i}) ∧ …
∧(¬z_{k_{i}−3}^{i} ∨ ℓ_{k_{i}−1}^{i} ∨ ℓ_{k_{i}}^{i}),

where z_{1}^{i}, …, z_{k_{i}−3}^{i} are new variables.

Let

φ′ := C′_{1} ∧ C′_{2} ∧ … ∧ C′_{k}.

Then φ′ is an input for 3SAT , and (6.11) holds.

Theorems 6.5.6, 6.5.8, 6.5.11, and 6.5.14 imply:

Theorem 6.5.15 The language Clique is NP-complete.

The traveling salesperson problem

We are given two positive integers k and m, a set of m cities, and an integer m × m matrix M, where

M(i, j) = the cost of driving from city i to city j,

for all i, j ∈ {1, 2, …, m}. We want to decide whether or not there is a tour through all cities whose total cost is less than or equal to k. This problem is NP-complete.

Bin packing

We are given three positive integers m, k, and ℓ, a set of m objects having volumes a_{1}, a_{2}, …, a_{m}, and k bins. Each bin has volume ℓ. We want to decide whether or not all objects fit within these bins. This problem is NPcomplete.

Here is another interpretation of this problem: We are given m jobs that need time a_{1}, a_{2}, …, a_{m} to complete. We are also given k processors, and an integer ℓ. We want to decide whether or not it is possible to divide the jobs over the k processors, such that no processor needs more than ℓ time.

Time tables

We are given a set of courses, class rooms, and professors. We want to decide whether or not there exists a time table such that all courses are being taught, no two courses are taught at the same time in the same class room, no professor teaches two courses at the same time, and conditions such as “Prof. L. Azy does not teach before 1pm” are satisfied. This problem is NP-complete.

Motion planning

We are given two positive integers k and ℓ, a set of k polyhedra, and two points s and t in ℚ^{3}. We want to decide whether or not there exists a path between s and t, that does not intersect any of the polyhedra, and whose length is less than or equal to ℓ. This problem is NP-complete.

Map labeling

We are given a map with m cities, where each city is represented by a point. For each city, we are given a rectangle that is large enough to contain the name of the city. We want to decide whether or not these rectangles can be placed on the map, such that

• no two rectangles overlap,

• For each i with 1 ≤ i ≤ m, the point that represents city i is a corner of its rectangle.

This problem is NP-complete.

This list of NP-complete problems can be extended almost arbitrarily: For thousands of problems, it is known that they are NP-complete. For all of these, it is not known, whether or not they can be solved efficiently (i.e., in polynomial time). Collections of NP-complete problems can be found in the book

• M.R. Garey and D.S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman, New York, 1979,

and on the web page

         http://www.nada.kth.se/~viggo/wwwcompendium/

194 CHAPTER 6. COMPLEXITY THEORY

6.6 Exercises

6.1 Prove that the function F : ℕ → ℕ, defined by F (x) := 2^{x}, is not in FP. 6.2 Prove Theorem 6.5.3. 6.3 Prove that the language Clique is in the class NP. 6.4 Prove that the language 3SAT is in the class NP. 6.5 We define the following languages:

• Sum of subset:

SOS := {⟨a_{1}, a_{2}, …, a_{m}, b⟩ : ∃I ⊆ {1, 2, …, m}, ∑_{i∈I} a_{i} = b}.

• Set partition:

SP := {⟨a_{1}, a_{2}, …, a_{m}⟩ : ∃I ⊆ {1, 2, …, m}, ∑_{i∈I} a_{i} = ∑_{i∉I} a_{i}}.

• Bin packing: BP is the set of all strings ⟨s_{1}, s_{2}, …, s_{m}, B⟩ for which 1. 0 < s_{i} < 1, for all i,

2. B ∈ ℕ,

3. the numbers s_{1}, s_{2}, …, s_{m} fit into B bins, where each bin has size one, i.e., there exists a partition of {1, 2, …, m} into subsets I_{k}, 1 ≤ k ≤ B, such that ∑_{i∈I_{k}} s_{i} ≤ 1 for all k, 1 ≤ k ≤ B. For example, ⟨1/6, 1/2, 1/5, 1/9, 3/5, 1/5, 1/2, 11/18, 3⟩ ∈ BP, because the eight fractions fit into three bins:

1/6 + 1/9 + 11/18 ≤ 1, 1/2 + 1/2 = 1, and 1/5 + 3/5 + 1/5 = 1.

1. Prove that SOS ≤_{P} SP.

2. Prove that the language SOS is NP-complete. You may use the fact that the language SP is NP-complete.

3. Prove that the language BP is NP-complete. Again, you may use the fact that the language SP is NP-complete.

6.6 Prove that 3Color ≤_{P} 3SAT .

Hint: For each vertex i, and for each of the three colors k, introduce a Boolean variable x_{ik}.

6.7 The (0, 1)-integer programming language IP is defined as follows:

IP := {⟨A, c⟩ : A is an integer m × n matrix for some m, n ∈ ℕ,

c is an integer vector of length m, and

∃x ∈ {0, 1}^{n} such that Ax ≤ c (componentwise) }.

Prove that the language IP is NP-complete. You may use the fact that the language SOS is NP-complete.

6.8 Let φ be a Boolean formula in the variables x_{1}, x_{2}, …, x_{m}. We say that φ is in disjunctive normal form (DNF) if it is of the form

φ = C_{1} ∨ C_{2} ∨ … ∨ C_{k},  (6.12)

where each C_{i}, 1 ≤ i ≤ k, is of the following form:

C_{i} = ℓ_{1}^{i} ∧ ℓ_{2}^{i} ∧ … ∧ ℓ_{k_{i}}^{i}.

Each ℓ_{j}^{i} is a literal, which is either a variable or the negation of a variable. We say that φ is in conjunctive normal form (CNF) if it is of the form

φ = C_{1} ∧ C_{2} ∧ … ∧ C_{k},  (6.13)

where each C_{i}, 1 ≤ i ≤ k, is of the following form:

C_{i} = ℓ_{1}^{i} ∨ ℓ_{2}^{i} ∨ … ∨ ℓ_{k_{i}}^{i}.

Again, each ℓ_{j}^{i} is a literal.

We define the following two languages:

DNFSAT := {⟨φ⟩ : φ is in DNF-form and is satisfiable},

and CNFSAT := {⟨φ⟩ : φ is in CNF-form and is satisfiable}.

1. Prove that the language DNFSAT is in P.

2. What is wrong with the following argument: Since we can rewrite any Boolean formula in DNF-form, we have CNFSAT ≤_{P} DNFSAT . Hence, since CNFSAT is NP-complete and since DNFSAT ∈ P, we have P = NP.

3. Prove directly that for every language A in P, A ≤_{P} CNFSAT . “Directly” means that you should not use the fact that CNFSAT is NPcomplete.

6.9 Prove that the polynomial upper bound on the length of the string y in the definition of NP is necessary, in the sense that if it is left out, then any decidable language would satisfy the condition.

More precisely, we say that the language A belongs to the class D, if there exists a language B ∈ P, such that for every string w,

w ∈ A ⟺ ∃y : ⟨w, y⟩ ∈ B.

Prove that D is equal to the class of all decidable languages.

0  1  (z_{1}, 1)  0  0  □ (z_{0}, 0)  1  0  1  1  □ # # # # # # # # #  # # # # # # # # #

Figure 6.7: The frame of the domino game for input x = 11.

198 CHAPTER 6. COMPLEXITY THEORY

[figure: 0; 1; (z_{1}, 1); 0; 0; □; (z_{0}, 0); 1; 0; 1; 1; □; #; #; #; #; #; #; #; #; #; #; #; #; #; #; #; #; #; #; #; #; 0; 0; #; #; 1; 1; #; #; (z_{1}, 1); (z_{2}, 0); #; #; 0; 0; #; #; 0; 0; #; #; □; □; #; #; 0; 0; #; #; 1; 1; #; z_{2}; (z_{2}, 0); 0; z_{2}; #; 0; (z_{2}, 1); #; #; 0; 0; #; #; □; □; #; #; 0; 0; #; #; 1; 1; #; #; 0; 0; #; z_{2}; (z_{2}, 1); 1; z_{2}; #; 0; (z_{2}, 1); #; #; □; □; #; #; 0; 0; #; #; 1; 1; #; #; 0; 0; #; #; 1; 1; #; z_{2}; (z_{2}, 1); 1; z_{2}; #; □; (z_{0}, □); #; #; 0; 0; #; #; 1; 1; #; #; 0; 0; #; #; 1; 1; #; z_{0}; 1; (z_{0}, 1); z_{0}; #; (z_{0}, □); □; #; #; 0; 0; #; #; 1; 1; #; #; 0; 0; #; z_{0}; 1; (z_{0}, 1); z_{0}; #; (z_{0}, 1); 1; #; #; □; □; #; #; 0; 0; #; #; 1; 1; #; z_{0}; 0; (z_{0}, 0); z_{0}; #; (z_{0}, 1); 1; #; #; 1; 1; #; #; □; □; #; #; 0; 0; #; z_{0}; 1; (z_{0}, 1); z_{0}; #; (z_{0}, 0); 0; #; #; 1; 1; #; #; 1; 1; #; #; □; □; #; z_{0}; 0; (z_{0}, 0); z_{0}; #; (z_{0}, 1); 1; #; #; 0; 0; #; #; 1; 1; #; #; 1; 1; #; #; □; □]

Figure 6.8: The solution of the domino game for input x = 11.

Summary

In this course, we have seen several different models for “processing” languages, i.e., processing sets of strings over some finite alphabet. For each of these models, we have asked the question which types of languages can be processed, and which type of languages cannot be processed. In this final chapter, we give a brief summary of these results.

Regular languages: This class of languages was considered in Chapter 2. The following statements are equivalent:

1. The language A is regular.

2. There exists a deterministic finite automaton that accepts A.

3. There exists a nondeterministic finite automaton that accepts A.

4. There exists a regular expression that describes A.

This claim was proved by the following conversions:

1. Every nondeterministic finite automaton can be converted to an equivalent deterministic finite automaton.

2. Every deterministic finite automaton can be converted to an equivalent regular expression.

3. Every regular expression can be converted to an equivalent nondeterministic finite automaton.

200 CHAPTER 7. SUMMARY

We have seen that the class of regular languages is closed under the regular operations: If A and B are regular languages, then

1. A ∪ B is regular,

2. A ∩ B is regular,

3. AB is regular,

4. A^{∗} is regular, and

5. \overline{A} is regular. Finally, the Pumping Lemma for Regular Languages gives a property that

every regular language possesses. We have used this to prove that languages such as {a^{n}b^{n} : n ≥ 0} are not regular. Context-free languages: This class of languages was considered in Chapter 3. We have seen that every regular language is context-free. Moreover, there exist languages, for example {a^{n}b^{n} : n ≥ 0}, that are context-free, but not regular. The following statements are equivalent:

1. The language A is context-free.

2. There exists a context-free grammar whose language is A.

3. There exists a context-free grammar in Chomsky normal form whose language is A.

4. There exists a nondeterministic pushdown automaton that accepts A. This claim was proved by the following conversions:

1. Every context-free grammar can be converted to an equivalent contextfree grammar in Chomsky normal form.

2. Every context-free grammar in Chomsky normal form can be converted to an equivalent nondeterministic pushdown automaton.

3. Every nondeterministic pushdown automaton can be converted to an equivalent context-free grammar. (This conversion was not covered in class.)

Nondeterministic pushdown automata are more powerful than deterministic pushdown automata: There exists a nondeterministic pushdown automaton that accepts the language

{vbw : v ∈ {a, b}^{∗}, w ∈ {a, b}^{∗}, |v| = |w|},

but there is no deterministic pushdown automaton that accepts this language. (We did not prove this.)

We have seen that the class of context-free languages is closed under the union, concatenation, and star operations: If A and B are context-free languages, then

1. A ∪ B is context-free,

2. AB is context-free, and

3. A^{∗} is context-free.

However,

1. the intersection of two context-free languages is not necessarily contextfree, and

2. the complement of a context-free language is not necessarily contextfree.

Finally, the Pumping Lemma for Context-Free Languages gives a property that every context-free language possesses. We have used this to prove that languages such as {a^{n}b^{n}c^{n} : n ≥ 0} are not context-free.

The Church-Turing Thesis: In Chapter 4, we considered “reasonable” computational devices that model real computers. Examples of such devices are Turing machines (with one or more tapes) and Java programs. It turns out that all known “reasonable” devices are equivalent, i.e., can be converted to each other. This led to the Church-Turing Thesis:

• Every computational process that is intuitively considered to be an algorithm can be converted to a Turing machine.

Decidable and enumerable languages: These classes of languages were considered in Chapter 5. They are defined based on “reasonable” computational devices, such as Turing machines and Java programs. We have seen that

1. every context-free language is decidable, and

2. every decidable language is enumerable.

Moreover,

1. there exist languages, for example {a^{n}b^{n}c^{n} : n ≥ 0} that are decidable, but not context-free,

2. there exist languages, for example the Halting Problem, that are enumerable, but not decidable,

3. there exist languages, for example the complement of the Halting Problem, that are not enumerable.

In fact,

1. the class of all languages is not countable, whereas

2. the class of all enumerable languages is countable.

The following statements are equivalent:

1. The language A is decidable.

2. Both A and its complement \overline{A} are enumerable.

Complexity classes: These classes of languages were considered in Chapter 6.

1. The class P consists of all languages that can be decided in polynomial time by a deterministic Turing machine.

2. The class NP consists of all languages that can be decided in polynomial time by a nondeterministic Turing machine. Equivalently, a language A is in the class NP, if for every string w ∈ A, there exists a “solution” s, such that (i) the length of s is polynomial in the length of w, and (ii) the correctness of s can be verified in polynomial time.

The following properties hold:

1. Every context-free language is in P. (We did not prove this).

2. Every language in P is also in NP.

3. It is not known if there exist languages that are in NP, but not in P.

4. Every language in NP is decidable. We have introduced reductions to define the notion of a language B to be “at least as hard” as a language A. A language B is called NP-complete, if

1. B belongs to the class NP, and

2. B is at least as hard as every language in the class NP. We have seen that NP-complete exist.

The figure below summarizes the relationships among the various classes of languages.

204 CHAPTER 7. SUMMARY

regular context-free P NP decidable enumerable all languages