I have a case where I need to choose a random subset of a list of know size $n$, but I don't know in advance how many items I will need (i.e. keep picking new indexes until $k$ are found that pass some test). What I want is an iterator that, once seeded with a random number, will generate a permutation of $[0, n-1]$ with space and time complexity significantly less than $O(n)$, ideally not greater than $O(k)$.
I've found a bunch of way to generate a single "random permutation" (LFSR, modular exponentiation, etc.) or a small number of them, but I'm not seeing how to re-work them into choosing one of a large set of permutations. And I need that because I do this repeatedly and $n$, $k$ and the expected "test pass rate" are such that I know I will eventually test all values and I want to avoid strong correlations between the inclusion/exclusion of any pair of elements. Off hand, I suspect I need something that can generate at least $n$ different permutations (where rotations are considered equivalent), but I'd prefer a solution with a range of size more like $n^2$.
The baseline solution would of course be to just shuffle a set of indexes, but that's $O(n)$ and $O(n \cdot log n)$ in space and time respectively which would work for the current size problem ($O(10^3)$), but I expect to become untenable within the expected future cases ($O(10^9)$).
