Let $P$ be an arbitrary random fixed permutation of $n$ elements. Each of the desired shuffle can be generated from the previous as follows:
- apply $P$ to the previous shuffle (or the initial set/vector, the first time);
- rotate the result one to the left;
- apply $P^{-1}$ to the result.
This will have the property asked by "if I pick a number at a particular position in the list after each shuffle, I should never get a duplicate". Proof: in applying the shuffle sequentially, the next application of $P$ with cancel the previous application of $P^{-1}$. It follows that the rotation will operate on the result of the previous rotation, thus that the property is met at the output of the rotation. It follows that the property is met after applying $P^{-1}$, thus at the output.
We can iterate this $n-1$ times (if we do it once more, we are back to the original, thus will get something clearly sequential, thus in the context distinguishable from the required "random integers" with great ease and near certainty).
We can use the Fisher-Yates shuffle to generate $P$. Or, if keeping it in memory is an issue because of size, we can use a cipher to implement $P$, and its reverse; that's a standard trick of Format Preserving Encryption.
Any isolated shuffle is indistinguishable from a random shuffle, but even the first two shuffles generated are distinguishable from random shuffles with the required property (which perhaps is undesirable, if not clearly prohibited).
If we remove the requirement that the same operation is iterated to go from one shuffle to the next (we are told that we can), we can do something better: pick three arbitrary random permutations $P$, $Q$, $R$ of $n$ elements, and get the $(j+1)^\text{th}$ shuffle as
- apply $P$ to the initial state
- rotate the result $Q(j)$ times to the left, where $Q(j)$ is the $(j+1)^\text{th}$ element of the vector obtained by applying $Q$ to the vector of the first $n$ non-negative integers; or, otherwise said, $Q(j)$ is an integer $q_j$ with $0\le q_j<n$, such that $0\le i<j<n\implies q_i\ne q_j$.
- apply $R$ to the result.
This time, we can obtain $n$ individually random-looking shuffles, which obviously is the maximum. Again, Fisher-Yates or FPE can be used to implement $P$, $Q$ and $R$. Even knowing the initial shuffle and the rank of any two generated shuffles, I vaguely conjecture (without proof) that they are indistinguishable from two random shuffles meeting the property asked. By a counting argument, three generated shuffles are distinguishable, at least by an unbounded adversary; thus we are far from generating the best randomness possible.
I have asked the question in more academic terms there.
Here is sample code in Java for the second method. Class MyCipher implements a Pseudo Random Permutation of size elements, made using cycling (arguably the simplest efficient FPE technique), and a basic iterated block cipher. Class MyShuffledArray is then a straightforward implementation of the technique that I propose. The code is usable for large parameters, and uses constant memory.
import java.util.Random;
import java.security.SecureRandom;
public class MyShuffledArrayDemo {
// MyCipher implement a PRP of size elements
static class MyCipher {
private static final int R = 40; // number of rounds
private final int size; // size of PRP
private final int mask; // bit mask for block
// 0 < size <= mask+1 and mask+1 is a power of 2
private final int rish; // right shift count
private final int[] rk = new int[R]; // rk for each round
// Constructor from rng source
private MyCipher(int size, Random rng) {
assert size>0 : "MyCipher.size must be positive";
int i,j;
// find block cipher width j in bits; and mask
for (j = 3, i = 8; j<31 && i<size; ++j)
i += i;
this.size = size;
this.mask = i-1; // one less than a power of two
this.rish = j*3/7; // shift count, at least 1
i = R; do
rk[--i] = rng.nextInt();
while (i!=0);
}
// Implement PRP, using a basic iterated block cipher and cycling.
// Input and output are a non-negative integer less than size.
private int Perm(int x) {
assert x>=0 && x<this.size : "bad input to MyCipher.encrypt";
do { // cycling loop; executed on average less than twice when size>4
int r = R;
do {// Round loop; each losslessly transforms x by
// - multiplying by 0xADB modulo a power of 2
// - adding a round key
// - XORing x with a right-shifted version
// Here, x<= mask and mask+1 is a power of 2
x = (x*0xADB+this.rk[--r]) & this.mask;
x ^= x>>>this.rish;
}
while (r!=0);
}
while (x>=this.size);
return x;
}
} // class MyCipher
// MyShuffledArray implement a virtual square array where each line and column appears to be a random permutation
static class MyShuffledArray {
private final int size; // dimension
private MyCipher P,Q,R;
// Constructor from rng source
private MyShuffledArray(int size, Random rng) {
assert size>0 && size<=0x40000000: "MyShuffledArray.size must be positive and at most 1073741824";
this.size = size;
P = new MyCipher(size, rng);
Q = new MyCipher(size, rng);
R = new MyCipher(size, rng);
}
// Implement the virtual array
private int Get(int col, int lin) {
return R.Perm((P.Perm(col)+Q.Perm(lin))%this.size);
}
} // class MyShuffledArray
// Example use
final static int START_OF_RANGE = 1;
final static int END_OF_RANGE = 9;
final static int NUMBER_OF_LISTS = 3;
final static int size = END_OF_RANGE - START_OF_RANGE + 1;
public static void main(String[] args) {
MyShuffledArray myShuffledArray = new MyShuffledArray(size, new SecureRandom());
for (int j = 0; j <NUMBER_OF_LISTS; j++) {
for (int i = 0; i <size; i++)
System.out.print(String.format(i==0?"[%d":", %d", myShuffledArray.Get(i,j) + START_OF_RANGE));
System.out.println("]");
}
}
}
This answer identifies that we are generating a latin square. Here is a literature survey.
