We know we haven't been duped because we utilize axioms instead of writing things down explicitly.
For instance, the natural numbers $0, 1, 2, \ldots$ ($\mathbb{N}$) are not constructed by writing out each integer in turn. Instead, we write out axioms: for all $x$, $x$ has a successor $Sx$; if $Sx = Sy$, then $x = y$; $0$ exists; for all nonzero $x$, $x = Sy$ for some $y$. And finally, for all $x$, $Sx \ne 0$.
Given just these simple axioms, we can be sure -- not concretely, but at least logically -- that we don't have $\mathbb{Z} / p \mathbb{Z}$. Because in arithmetic mod $p$, $S(p-1) = 0$, which we have asserted is not the case for any $x$.
In other words, it's a theorem following from the axioms that we aren't working in $\mathbb{Z} / p \mathbb{Z}$.
Even if you we can't write it down explicitly for every $p$.
Unless you object to the very idea of logical abstraction, it's hard to disagree with the above argument. If you accept the plausible premises about the successor function, then you must accept the conclusions. Yes, it's a bit alarming that we can prove things about integers that we could never even hope to write down, but on the other hand, this sort of logical generality is exactly what gives mathematics most of its power.
(On the other hand, Qiaochu points out in the comments that you could also disagree with the assumption that the axioms about the successor function $S$ are consistent. And there are probably numerous other ultrafinitist responses and philosophical nuances related to this.)
The bad news is, that while we can prove we aren't working in $\mathbb{Z} / p \mathbb{Z}$, we can't prove that we are actually working in $\mathbb{N}$, no matter how many axioms we add.
We could instead be working in a different infinite set, $S$, that "looks exactly like" $\mathbb{N}$ but isn't.
By "looks exactly like" $\mathbb{N}$", I mean that you could add any number of axioms you like--even an infinite number of axioms--and if they are all satisfied by the usual natural numbers $\mathbb{N}$, then they are satisfied by this imposter set $S$. So there is no hope of ever distinguishing $\mathbb{N}$ from the set $S$, yet $S$ is not equal to $\mathbb{N}$.
In fact, there are such imposter sets $S$--called "models of $\mathbb{N}$"--of any infinite cardinality you like. This is known as the Lowenheim-Skolem theorem, and is one of the most fundamental results in model theory. Interestingly, Skolem himself was somewhat of a finitist; he originally proposed the theorem and other results as an attack against the infinite, and in particular against the idea of there being differing infinite cardinatlies. Today, his "paradox" is instead accepted as a "theorem".
One unrelated note: while it is clear what you mean when you say $p$ is "uncomputably" large, this isn't usually what uncomputable means. In fact, any finite number is considered computable, on its own. Only a function $f: \mathbb{N} \to \mathbb{N}$ can be uncomputable; it has to be impossible to write a finite program that would output $f(x)$ on any input $x$. Anyway, I think we do need more terminology in mathematics to deal with your idea of "uncomputable", i.e. too large to actually write down given some finite set of memory $M$. Perhaps this terminology exists and I am just unaware of it.
