3

Say you have some unknown combination of $K$ different (independent) numbers you have to guess. Your goal is to guess this entire combination in order. Say K=16, you have to guess $k_1, ..., k_{16}$. Each number $k$ is limited to $N$ possible numbers (the cardinality of the set from which each number is chosen). Assume $N$ is the same for every $k$, i.e. all numbers are picked from identical sets. To be clear, since each of the sixteen numbers is picked independently, some of the secret numbers may be identical.

To guess each number $k$, if you assume all values are equally likely, you start with a list of the $N$ possible numbers, one of which is the correct number. You test and eliminate each at a time. For each of these $k \in K$, you already know the expectation of the number of guesses to be $E[p_k,\star] = \frac{N+1}{2}$ from the result in "Expected number of draws until the first good element is chosen". (the notation $p_k$ means that this is the position in list which you reach when guessing the correct number, denoted by $\star$).

Note that, a priori, you cannot verify that any of the $k$ numbers is correct, until you have the entire combination. However you may know the expectation of finding the correct value for each of $k$ numbers, since they are independent problems.

Now, if one expects to have to guess $\frac{N+1}{2}$ times for each $k$, the combinations of all these expected guesses for the total combination yields $E[p,\star] = {(\frac{N+1}{2})}^{16}$.

However, the same reasoning could also be applied to the combination of the independent numbers. So now, let's apply this formula to the entire combination, which is an equivalent problem.

If each independent number $k$ has $N$ possibilities, there are a total of $M = N^{16}$ possible combinations. If one has a list of all the possible $M$ combinations, one can expect to guess the correct combination of 16 numbers after $E[p,\star] = {(\frac{M+1}{2})}$.

However, trivially, ${(\frac{N+1}{2})}^{16} \neq {(\frac{N^{16}+1}{2})}$, and the error can grow quite big for bigger N.

Now say that the expectation for each $k$ is not $\frac{N+1}{2}$, it is instead some value that I approximated experimentally - perhaps I have some algorithm that makes it so I do not need $\frac{N+1}{2}$ guesses, but actually less because the process isn't as random as it claimed. This algorithm may have different expectations for each $k$, since it may not perform as well guessing some of the numbers.

How could I derive, generally, the expectation of the whole combination from the expectation of the parts, when only the whole combination can be verified?

Update: I realized I could not possibly define the expected position of the complete combination without defining a strategy. So, say that the strategy is: start by testing the combination of the best guess for each $k$, then if that is unsuccessful, test all the guesses for the $k$ with the highest expectation $E[p_k,\star]$, since that is the one with most uncertainty, and then proceed by testing all the remaining $k$ in decreasing order of expectation.

Especially Lime
  • 46,692
Ricardo M.
  • 43

2 Answers2

1

I don't think that it is possible to derive the expectation of the whole only from the parts. The expectations of the parts don't contain enough information about the probability distribution. Here is a counter-example:

Suppose you have to guess 2 numbers, each picked from the set $\{1, 2, 3\}$. The first one is picked with probabilities $(0.4, 0.4, 0.2)$, while the second one is picked with probabilities $(0.6, 0.2, 0.2)$. Independently, the expectations will be $1.8$ for the first and $1.6$ for the second. If you write out all the probabilities for the 9 possible combinations, and guess them in decreasing order (which is the best strategy), you'll get an expectation of $3.52$.

Now change the distribution on the second number to $(0.5, 0.4, 0.1)$. Independently, the expectation for that is the same as before, $1.6$. But as a combination with the first number (with the same strategy as before), I now get an expectation of $3.48$.

So the overall expectation is not a function of only the expectations of the parts.

Of course, my example uses a different strategy from the one you prescribed in the update to your question. When I thought more about it, I found that your strategy does give a way to "combine" the expectations (but as a strategy, it is sub-optimal). The formula I got is (assuming the expectations are ordered from highest to lowest) $$E[p, \star] = 1 + \sum_{i = 1}^{K}N^{i-1}(E[p_i,\star] - 1)$$ I think the easiest way to prove it is by induction on $K$. With my examples above, it gives $3.6$, which is not much worse than what the optimal strategies produce.

Nick Pavlov
  • 1,583
  • Thank you for this! I have managed to understand how you got your result (the -1 and adding +1 after threw me off a bit at first). While you showed it was impossible to derive the expectation of the total from the parts, using the(not optimal) strategy I am now able to get an upper bound. This has inspired me to improve the original strategy by instead of using the expectation of each part, I could use some percentile, such as the 99-percentile of the parts. The 99-percentile of each part will yield combinations that I assume lead to the 85-percentile of the total $(0.99)^{16} \approx 0.85$? – Ricardo M. Sep 29 '17 at 16:27
  • By the way, your reasoning has allowed me to actually be able to picture and calculate the number of guesses when replacing the brute-force with the percentile case, simply by replacing $N^{i-1}$ by the number of combinations that were tested before. – Ricardo M. Sep 29 '17 at 16:35
  • Well, I'm not sure what improvements on the strategy you can accomplish this way; I can only clearly understand why the strategy is flawed, and its inefficiency increases with increasing $N$, because you would be guessing a large number of less likely combinations. I can't say that I follow your percentile idea very clearly. It looks to me like you are imposing a cut-off on tries within each part, and that can lead you to an "expected number of guesses to achieve a given success rate" (such as 85%) but I don't see what good that is. – Nick Pavlov Sep 29 '17 at 19:00
  • A case which illustrates this is if the the set to pick from ($N$) is actually infinite, but with a finite expectation value (because the probability of a number being correct could decrease quickly enough). Then, the brute-force strategy will not even give you a finite expectation. With a hard cut-off, you can get an expectation for 85% success, but the true expectation could be many times greater than that (in fact, the percentile doesn't provide any kind of upper bound for it). – Nick Pavlov Sep 29 '17 at 19:06
  • Sorry, I wasn't clear. The upper bound refers to the brute-force strategy (which in your extreme case, being infinite for an infinite set, is clearly so). We have already reasonably concluded that one could not get the expectation of the optimal strategy, but we can get the expectation for a strategy that will ensure we get the correct combination for some percentage of the time. – Ricardo M. Sep 29 '17 at 19:40
  • In my case, I'm characterising a crypto attack. If, for some algorithm, I get some large expectation for 99% percentage, combined with an infinite brute-force expectation as the upper bound, I can already compare it with some other algorithm. The brute-force strategy is my "worst case" for that algorithm and the 99% case may not directly tell me the true expectation, but works as a metric for comparison between different algorithms. As for the optimal strategy, if it depends on the probability distributions, those change with every single attack run even in the same conditions,so no luck there – Ricardo M. Sep 29 '17 at 19:46
  • So, in conclusion, I will actually always use the optimal strategy when possible to guess my $k$ numbers. The other two strategies just help me gauge how well my partial guesses help my combination guess. Since I never actually have the real distribution of the possible values for the numbers I am guessing - only estimates at best, which vary in every case even if sampled in the same conditions - I have no direct way of estimating the true expectation. I could try estimating it experimentally, but as for my question, I believe that's as far as I can go. – Ricardo M. Sep 29 '17 at 20:03
  • OK. But just one final thought - if, for your algorithm running on an individual $k$ number, you have a way of evaluating any partial expectation (any percentile, not just 99th), that actually does contain all the information to devise a near-optimal strategy, I think. At least for when $N$ is large enough so that you can approximate expectations with an integral. Essentially it would encode the relevant info about the distributions. – Nick Pavlov Sep 29 '17 at 20:56
  • Is there anything I can read on this? I have an intuition of what you are suggesting - different percentiles roughly outline the spread of the correct partial values, heck, I could trivially get the median by definition. My intuition is not enough for any actionable knowledge though. – Ricardo M. Sep 29 '17 at 21:09
  • There probably is, but I don't know where. My expertise in this topic is not from formal training, certainly not at the level you need. I work these things out from scratch when I need to, but that's also why my insight is more theoretical than of practical use. The claim I made in the last comment is not something I can rigorously substantiate off the top of my head (hence the "I think" at the end of that sentence). – Nick Pavlov Sep 29 '17 at 21:35
0

The expected number of guesses with the first method is by far lower than the one you got. This is because you have to sum - not multiply - the expected values of the number of guessings for the individual numbers. Then from linearity of expectations we have $$E[p, \: \star] = \sum_{k = 1}^{16} E[p_k, \: \star] = 16\left(\frac{N + 1}{2}\right) = 8(N + 1)$$

For the second method the expectation you found is correct.

So the two expectations do differ, and the first one is actually much much less than the second one (the former grows linearly with $N$, the latter is a polynomial of degree $16$).

Why? Because guessing one number at a time gradually gives you some information about the combination (e.g. after having guessed the first number you know for sure that the combination isn't among the ones that don't start with a different number, so that you won't even try to guess those ones), which doesn't happen when you blindly try to guess the whole sequence.

cip999
  • 2,016
  • Hi, the problem is, I do not actually have that information - i.e., I can only say for sure that the entire combination is correct. Though I may know each of the parts to be correct with some degree of confidence, I cannot assume it is correct. What is happening is that the formula I provided for the individual numbers is not accurate since I actually have some insight into which numbers may have been chosen. I may know the expectation of each number individually, perhaps experimentally, by running guessing games where I know each of the individual numbers. I can only verify an entire guess. – Ricardo M. Sep 24 '17 at 16:12
  • Okay, I misunderstood what you meant... thanks for explaining. But then how would you assume that $E[p, : \star] = \prod E[p_k, : \star]$ (because that's what you did, isn't it?)? I can't see any reason why that should be true. – cip999 Sep 24 '17 at 16:21
  • Yes, and I had an intuition, but my question was precisely because I wasn't convinced due to the inequality I showed. My assumption was that, if I had to test on average, say, 2 guesses for $k_1$ and 4 guesses for $k_2$, then I would have to test 2*4=8 guesses on average for a combination $k_1, k_2$, since those are the possible permutations for that combination. – Ricardo M. Sep 24 '17 at 17:18