Why do we use the number of compares to measure the time complexity when compare is quite cheap?

Question

I think one reason a compare is regarded as quite costly is due to the historical research as remarked by Knuth, that it came from tennis match trying to find the second or third best tennis player correctly, assuming the tennis players are not a "rock paper scissors" situation (but has an "absolute 'combat' power").

If we have an array of size n being 1,000,000, we don't usually mind comparing 2,000,000 times to find the second largest number. With the tennis tournament, having a Player A match with Player B can be costly as it can take a whole afternoon.

With sorting or selection algorithms, for example, what if the number of comparisons can be O(n log n) or O(n), but then, other operations had to be O(n²) or O(n log n), then wouldn't the higher O() still override the number of comparisons? (Maybe it didn't happen yet, or else we would have a study case about this situation). So ultimately, shouldn't the number of atomic steps, instead of comparisons, measured by the order of growth as compared to n (O()) that determines the time complexity?

Answer 1

Sure. But in practice that is rare: the sorting algorithms we usually use or analyze in practice do at most a constant number of other operations per comparison, so this isn't an issue for the sorting algorithms we actually care about. This means that measuring the number of comparisons or the number of steps taken gives you the same asymptotic running time.

Also, there are some situations where we use sorting algorithms where each comparison is much slower than the other operations. A comparison may be slower than the other operations, because conditional branches can be very slow on modern processors (each comparison may have a significant chance to cause pipeline flushes). And if you are sorting complex objects with a custom comparison function, then each comparison might take many instructions. So, in at least some cases, the number of comparisons might well dominate the over time it takes to sort the input.

Finally, there are certainly some cases we do take into account the time to perform other operations. It's most common to only count the number of comparisons, but by no means universal. In situations where the time taken for other operations is important, people do analyze the total running time.

Ultimately, asymptotic running time analysis is only a theoretical model. It is a simplified model that omits many considerations. Any such simplification will necessarily ignore many factors. As long as those factors are unimportant, this can be useful, as it helps you simplify the problem enough to analyze it and gain insight. But, as always, if one of those factors turns out to have a significant effect and you didn't include it in your model, then the model will yield misleading results -- this is true of all models, and is not limited to sorting or algorithmic analysis. Part of the art of modelling is identifying which factors dominate and which factors are second-order, so that you can choose a model that's only as complex as needed -- it is as simple as possible, so you can understand it better, but no simpler than that, so it still yields predictions that are reasonably representative of the real world.

Answer 2

If moving an item were n times more expensive than comparing, selection sort would suddenly be the most efficient algorithm. But if moving an item were expensive, we could sort an array of array indices, and then sort the original array in place with at most 1.5n moves. (That would actually be n/2 swaps; if a swap is cheaper than 3 moves then worst case is 4/3 n).

The number of comparisons cannot be lower than log n! for comparison based sorting.

Answer 3

what if the number of comparisons can be O(n log n) or O(n), but then, other operations had to be O(n²) or O(n log n), then wouldn't the higher O() still override the number of comparisons?

Um, yeah. That's why, in such cases, we do use those other operations for the analysis. For example:

Binary insertion sort employs a binary search to determine the correct location to insert new elements, and therefore performs ⌈log₂ n⌉ comparisons in the worst case, which is O(n log n). The algorithm as a whole still has a running time of O(n²) on average because of the series of swaps required for each insertion.

Answer 4

Big Oh analysis is looking at the asymptotic behavior of an algorithm. In these analyses, the thing which gets done "more" will always overshadow the thing which gets done less. As an example, consider an algorithm which does $O(n)$ disk operations at a cost of 1,000,000,000 units each, and $O(n^3)$ comparisons at a cost of 1 unit each. Obviously for small numbers of n, the disk operations will be the dominating part of the cost. However, as we get to large n, like 1,000,000,000, we see that we do 1,000,000,000 disk operations, for a total cost of 1,000,000,000,000,000,000 and we do 1,000,000,000,000,000,000,000,000,000 comparisons, meaning we spent literally a billion times more on the comparisons than on the disk reads, even though they are very light weight on their own.

This is not always the story we want. For many practical algorithms, we are not operating on such awful huge datasets where those small comparisons start to add up. In these cases we may do other things. In the case of "offline algorithms," which operate on databases on disks, we may measure the number of reads. Or we may even recognize that its cheaper to load a "page" of values all at once, so we try to measure how many pages have to get loaded.

This sort of study is useful, but it is not well captured in a simple Big Oh analysis. It typically has to account for what the hardware is good at, making the analysis more specialized. For example, there are cryptographic algorithms which are designed to be very inefficient on a GPU architeture, but very efficient on a CPU architecture.

My personal favorite of these is a particular disjoint set algorithm that came up when I was working on some fun threading problems. The complexity of it was one of those exotic things with O(log log N) terms showing up. But in the conclusion of the paper, they had to admit that their algorithm had some aspects which had such an ungodly large time constant that the less advanced algorithms were better for basically any dataset which fit onto a modern disk farm. Once your dataset started to move into the extabyte size scale, their algorithm started to earn its salt!

Answer 5

It seems like none of the other answers so far have mentioned a basic reason: There are many sorting algorithms whose total (asymptotic) time complexity is in fact bounded by the number of comparisons times the maximum time taken per comparison. Obviously that is why such sorting algorithms' cost should be measured by the number of comparisons, because they can be used to sort any input list from any set given a comparison function on that set.

Another more theoretical reason is that the information theoretic lower bound for sorting applies to the number of comparisons in the comparison model, so of course it makes most sense to measure the cost of a generic sorting algorithm in terms of the number of comparisons it takes.

Answer 6

1. Because you can avoid the other operations being expensive.

Comparing integers is cheap. Comparing arbitrary elements of type T is typically as expensive as reading sizeof(T) bytes. Now, you might think "ok, but copying type-T elements is also expensive" - but we don't need to do that. We can just copy or move their indices in the input around and use those to lookup the original elements when comparing. Yet, we can't avoid the comparisons the same way.

Caveat: You can use hashing to avoid, typically, most costly comparisons.

2. It doesn't matter (except when it does)

You asked:

what if the number of comparisons can be O(n log n) or O(n), but then, other operations had to be O(n²) or O(n log n)

Well, the first option is a poor sorting algorithm anyway, and as for the second option - see @HeapOverflow's answer.

Answer 7

The cost of comparison is more complex and depends on more than you imagine. Between each of the following is an order of magnitude cost increase.

• Register
• CPU Cache
• RAM
• Magnetic disk

Then there's the fact that your process does not run in isolation, but competes with other processes for system resources. Some of those processes will not be denied.

In the world of database engines, performance is all about mitigation of working set size. The smaller your working set, the less time you spend in the Slow Zone.