Algorithmic intuition for logarithmic complexity

56

26

I believe I have a reasonable grasp of complexities like $\mathcal{O}(1)$, $\Theta(n)$ and $\Theta(n^2)$.

In terms of a list, $\mathcal{O}(1)$ is a constant lookup, so it's just getting the head of the list. $\Theta(n)$ is where I'd walk the entire list, and $\Theta(n^2)$ is walking the list once for each element in the list.

Is there a similar intuitive way to grasp $\Theta(\log n)$ other than just knowing it lies somewhere between $\mathcal{O}(1)$ and $\Theta(n)$?

Khanzor

Posted 2012-03-21T05:51:41.653

Reputation: 623

8log n is for "searching": think binary search – Suresh – 2012-03-21T06:07:50.300

2

Using $O$ to ask this question is incorrect, as it only denotes an upper bound. For instance constant time is $\mathcal{O}(\log n)$. $\theta$ would be more appropriate. See meta question: http://meta.cs.stackexchange.com/questions/182/editing-a-question-and-almost-all-the-answers

– Aryabhata – 2012-03-22T21:47:33.663

1

More information on SO: what does $O(\log n)$ mean exactly?.

– Ran G. – 2012-04-02T04:07:08.973

A little note: In the classical Turing Machine settings, all algorithms are $\Omega (n)$, since they need to read each symbol of the input at least once. Binary search can be done in $O(\log n)$ because we have the promise that the list is sorted, for example. – chazisop – 2012-06-19T13:38:30.703

1A late contribution: by definition, the base $b$ logarithm of a number $n$ is just the number of times you multiply $b$ by itself to get $n$. $b^l = n \iff l = log_b(n)$. For example, $2^3 = 8 \iff log_2(8) = 3$. So if you have a number $n$ and you want to find out what $log_b(n) = ?$ just keep dividing it by $b$ until you reach a $1$ (assuming $n$ is a power of $b$ for simplicity). The number of divisions is equal to $log_b(n)$. Algorithms that exhibit this division behavior have running times in $O(log(n))$. – saadtaame – 2012-08-20T03:13:44.657

Answers

53

The $\Theta(\log n)$ complexity is usually connected with subdivision. When using lists as an example, imagine a list whose elements are sorted. You can search in this list in $\mathcal{O}(\log n)$ time - you do not actually need to look at each element because of the sorted nature of the list.

If you look at the element in the middle of the list and compare it to the element you search for, you can immediately say whether it lies in the left or right half of the array. Then you can just take this one half and repeat the procedure until you find it or reach a list with 1 item which you trivially compare.

You can see that the list effectively halves each step. That means if you get a list of length $32$, the maximum steps you need to reach one-item list is $5$. If you have a list of $128 = 2^7$ items, you need only $7$ steps, for a list of $1024 = 2^{10}$ you need only $10$ steps etc.

As you can see, the exponent $n$ in $2^n$ always shows the number of steps necessary. Logarithm is used to "extract" exactly this exponent number, for example $\log_2 2^{10} = 10$. It also generalizes to list lengths that are not powers of two long.

Karel Petranek

Posted 2012-03-21T05:51:41.653

Reputation: 616

@AndrewMyers Searching a linked list is more accurately $\mathcal{O}(n)$ – phant0m – 2012-09-30T21:37:11.840

@phant0m Yes, I meant that the specific algorithm in the answer would be O(n log n) in the case of linked lists. – asm – 2012-10-01T11:58:54.360

@AndrewMyers binary search? I didn't see a different algorithm being mentioned. – phant0m – 2012-10-01T12:43:09.390

@phant0m Yes, Binary search is only O(log n) if you have O(1) random access right? So in the case of linked lists you have O(n) random access O(log n) times unless I'm badly mistaken. – asm – 2012-10-01T12:57:56.423

@AndrewMyers You can't just multiply random access time by log n to get the result. You cannot perform binary search on a normal linked list. In the worst case, you need to traverse the entire list: O(n) to search in a linked list. – phant0m – 2012-10-03T09:22:16.897

@phant0m No, your best case algorithm for a linked list is O(n), if you implement binary search on a linked list it will be O(n log n). There is no reason that you cannot perform binary search on a normal linked list it just doesn't make sense since there's a better algorithm. – asm – 2012-10-03T11:57:17.493

@AndrewMyers Ah, I understand what you mean now. But even if you were to perfom a binary search, the complexity is still O(n): You do something like $\sum_{i=0}^{\log_2 n - 1}2^i \approx n$ many steps in the worst case. – phant0m – 2012-10-03T14:41:24.837

@phant0m True, a smarter implementation is better. – asm – 2012-10-03T15:03:18.167

@AndrewMyers Oh, my formula is only accurate for doubly linked lists (and a smart implementation). You were right with $n \log n$ on singly lists with a "stupid" algorithm. – phant0m – 2012-10-03T15:04:47.110

1@phant0m Yeah, took me a bit to figure out that it's assuming you're moving from the current position instead of traversing from the beginning every time. – asm – 2012-10-03T15:06:18.490

4It should be noted that this is only O(log n) if the list has constant time random access. On more typical list implementations (linked lists) this is O(n log n) – asm – 2012-03-21T12:53:19.853

1Binary search does not work on lists for the lack of pointers; it is usually done on arrays. – Raphael – 2012-03-21T13:08:09.737

Binary search works just fine on lists. It's just quite pointless on account of being a lot more complicated than required/usefull/practical. – Anton – 2012-03-21T15:18:24.667

38

In terms of (balanced) trees (say, binary tree, so all $\log$'s are base 2):

  • $\Theta(1)$ is getting the root of the tree
  • $\Theta(\log n)$ is a walk from root to leaf
  • $\Theta(n)$ is traversing all nodes of the tree
  • $\Theta(n^2)$ is actions on all the subsets two nodes in the tree, e.g., the number of different paths between any two nodes.
  • $\Theta(n^k)$ - generalization of the above for any subset of $k$ nodes (for a constant $k$)
  • $\Theta(2^n)$ is actions on all the possible subsets of nodes (subsets of all the possible sizes, i.e., $k=1,2, \ldots, n$.). For instance, the number of different sub-trees of the tree.

Ran G.

Posted 2012-03-21T05:51:41.653

Reputation: 15 978

5To add to this, intuition for $\Theta(log\,log\,n)$ comes from two things: 1.) The recurrence $T(n) = T(\sqrt(n)) + 1$ and 2.) Binary search on something of size $log(n)$ i.e. a binary search on the height of the tree. – mcorley – 2012-03-25T16:49:01.580

17

For $O(\log n)$ to be possible, you need to be able to cut down the problem size proportionally by some arbitrary amount with respect to $n$ with a constant time operation.

For example, in the case of binary search, you can cut down the problem size by a half with each comparison operation.

Now, do you have to cut down the problem size by half, actually no. An algorithm is $O(\log n)$ even if it can cut down the problem search space by 0.0001%, as long as the percentage and the operation it uses to cut down the problem size stays constant, it's a $O(\log n)$ algorithm, it won't be a fast algorithm, but it's still $O(\log n)$ with a very large constant. (Assuming we are talking about $\log n$ with a base 2 log)

Ken Li

Posted 2012-03-21T05:51:41.653

Reputation: 1 423

1What would it be if the 'down cutting amount' wasn't constant? – Svish – 2012-03-21T15:04:05.750

@Svish If you can cut the problem down at a positive rate, it would still be a $O(\log n)$ algorithm, although it probably would no longer be a tight bound. Negative rate is hard to say. The assumption was made to make the answer rather simple in this case, since this question has its own merit, you are more than welcome to ask this as a question on its own. – Ken Li – 2012-03-21T21:42:40.790

Yeah, I was meaning that the problem search space always decreased, but not necessarily at a constant rate. Was just thinking of your "as long as the percentage and the operation it uses to cut down the problem size stays constant, it's a O(log n) algorithm"; if it had a different name if the percentage there wasn't constant. – Svish – 2012-03-21T22:15:07.740

8

Yes, $\log(n)$ is between $1$ and $n$, but it is closer to $1$ than $n$. What is $\log(n)$? The log function is the inverse function of exponentation. Let me start with exponentation and you should get a better idea of what logarithm is.

Consider two numbers, $100$ and $2^{100}$. $2^{100}$ is $2$ multiplied with itself $100$ times. You can with some effort count $100$ numbers, but can you count $2^{100}$? I bet you can't. Why? $2^{100}$ is such a big number that it is greater than the number of all atoms in the universe. Reflect on that for a moment. It is such a huge number, that it allows you to give each atom a name (number). And the number of atoms in your finger nail is probably in the order of billions. $2^{100}$ ought to be enough for anyone (pun intended :)).

Now, between the two numbers, $100$ and $2^{100}$, $100$ is the logarithm of $2^{100}$ (in base $2$). $100$ is comparatively such a small number than $2^{100}$. Anybody ought to have $100$ different items in their home. But, $2^{100}$ is good enough for the universe. Think home vs universe when thinking of $\log(n)$ and $n$.

Where do exponentation and logarithms come from? Why are they of so much interest in computer science? You may not notice, but exponentation is everywhere. Did you pay interest on credit card? You just paid a universe for your home (Not so bad, but the curve fits). I like to think that exponentation comes from product rule, but others are welcome to give more examples. What's product rule, you may ask; And I shall answer.

Say you have two cities $A$ and $B$, and there are two ways to go between them. What is the number of paths between them? Two. That is trivial. Now say, there is another city $C$, and you can go from $B$ to $C$ in three ways. How many paths are there between $A$ and $C$ now? Six, right? How did you get that? Did you count them? Or did you multiply them? Either way, it is easy to see that both ways give a similar result. Now if you add a city $D$ which can be reached from $C$ in four ways, how many ways are there between $A$ and $D$? Count if you don't trust me, but it is equal to $2\cdot 3\cdot 4$ which is $24$. Now, if there are ten cities and there are two paths from one city to the next, and they are arranged like they are on a straight line. How many paths are there from start to end? Multiply them if you don't trust me, but I will tell you there are $2^{10}$, which is $1024$. See that $2^{10}$ is exponential result of $10$, and $10$ is the logarithm of $2^{10}$. $10$ is a small number compared to $1024$.

The logarithm function $\log_2(n)$ is to $n$ what $n$ is to $2^n$ (note that $2$ is the logarithm's base). If you multipy $\log_b(n)$ with itself $b$ times (note that $b$ is the logarithm's base) you get $n$. $\log(n)$ is so tiny, so small compared with $n$, that it is size of your home where $n$ is size of the universe.

On a practical note, $\log(n)$ functions perform very similar to constant functions. They do grow with $n$, but they grow very slowly. If you optimized a program to run in logarithmic time which was taking a day before, you will probably run it in the order of minutes. Check for yourself with problems on Project Euler.

Ravi

Posted 2012-03-21T05:51:41.653

Reputation: 289

3While nicely written, this answer contains hardly any information besides "$\log(n)$ is really small". – Raphael – 2012-03-21T13:05:35.293

3I was trying to give an intuition for how small it is. – Ravi – 2012-03-21T21:36:40.050

8

Think about algorithm to convert decimal number $n$ to binary

while n != 0:
   print n%2, 
   n = n/2

This while loop runs $\log(n)$ times.

Pratik Deoghare

Posted 2012-03-21T05:51:41.653

Reputation: 1 011

1Certainly this program loops $\log n$ times, but generally when we talk about $O(f(s))$ complexity, $s$ is the size of your input. Here the size of your input is already $s=\log n$, so I would say this program is only linear (in $O(s)$) – jmad – 2012-03-21T14:02:16.830

@jmad Right. But this example does give you intuition into log(n). – Pratik Deoghare – 2012-03-21T14:05:37.793

@jmad I could have used algorithm to generate random numbers too but I wanted it as simple as possible.

– Pratik Deoghare – 2012-03-21T14:10:35.680

5

To continue your theme, $O(\log n)$ is like repeatedly guessing where $x$ lies in the list and being told "higher" or "lower" (in terms of index).

It's still based on the size of the list, but you only need to visit a fraction of the elements.

dpatchery

Posted 2012-03-21T05:51:41.653

Reputation: 151

3

If we have a divide and conquer algorithm, and we make only one recursive call for a subproblem, and it is the second case in Master theorem, i.e. the time complexity of the non-recursive part is $\Theta(\lg^k n)$, then the complexity of the algorithm will be $\Theta(\lg^{k+1} n)$.

In other words, when we have a divide and conquer algorithm with one recursive call to itself on a problem with size a constant factor of the current problem, and the time in the non-recursive part is $\Theta(1)$ (constant), then the running time of the algorithm is going to be $\lg n$.

The binary search algorithm is the classical example.

Kaveh

Posted 2012-03-21T05:51:41.653

Reputation: 16 741

1

The intuition is how many times you can halve a number, say n, before it is reduced to 1 is O(lg n).

For visualizing, try drawing it as a binary tree and count the number of levels by solving this geometric progression.

2^0+2^1+...+2^h = n

user1234

Posted 2012-03-21T05:51:41.653

Reputation: 41

Welcome to the site! What you say is, of course, true but I don't see what it adds to the existing answers. Several of the answers already say that the logarithm is the number of times you can divide by two before hitting 1, and Ran's answer already says that $\log n$ is the height of a binary tree with $n$ leaves. – David Richerby – 2016-12-28T19:57:08.307