[This post is an attempt to inform others on an interesting finding in Gauss's Nachlass; I know math stack exchange format is intended primarily for more "closed" answers, but I felt like it is my duty to acknowledge interested readers on this finding. This question looks seeks for an historical survey of the development p-adic numbers, while my answer focuses on its prehistory.]
In a recently scanned notebook of Gauss (Schaede 5), one can find a note whose title is "Congruentia infinita", which includes the following results (a pic of the relavent page is shown at the end of this answer):
Congruentia infinita $$x^5-20x^4-86x^3-98x^2+80x+3\equiv 0 \pmod {241^{\infty}}$$ has roots: $$(1) = 2+191.r+$$ $$(2) = 3+$$ $$(3) = 4+$$ $$(4) = 5+$$ $$(5) = 6+$$
and in the bottom of this page Gauss also writes:
$$\sqrt{5}\pmod{11^{\infty}}= 9.0.4.10.4.4$$
The last result is about the last digits of the square root of 5 calculated 11-adically, and according to this post, this result is actually correct (he apparently needed this square root in order to calculate the 11-adic expansion of the fifth root of unity). These results show that Gauss almost certainly preconceived of the p-adic numbers, since he used such expressions as $\pmod {11^{\infty}}$. It must be mentioned that since Kurt Hensel's first publication on this number system appeared only in 1897, the fact that Gauss anticipated it somehow escaped the attention of the editors of Gauss's collected works, so they did not publish this important fragment.
In the recent literature on the history of p-adic numbers, I found two references that "suspect" that Gauss had something similar to p-adic analysis in his arsenal:
- The article "on the origin of p-adic analysis" by Peter Ulrich, which mentions Gauss's proof of his lemma on polynomials (from article 42 of the D.A) using valuation theory as part of the prehistory of p-adic analysis.
- The article "The Unpublished Section Eight: On the Way to Function Fields
over a Finite Field" by Gunther Frei, which observes that Gauss discovered and proved an early version of Hensel's lemma, and makes the following speculation:
It looks almost as if Gauss thought of some kind of algebraic closure of finite fields or some kind of p-adic version of the complex numbers. Unfortunately he did not leave us more details of these ideas than what he started to treat in the §§ 373–375 of the Disquisitiones generales de congruentiis. After entry 80, Gauss occasionally continued to work on higher reciprocity laws, on the Fundamental Theorem of Algebra and on the theory of cyclotomy, but he never came back to the theory of polynomials over a finite field.
I noticed several things about Gauss's attempt to find the five roots of the quintic polynomial $\pmod {241^{\infty}}$:
- The first parts of the five roots all satisfy the polynomial $\pmod {241^1}$. It also seems interesting that in this particular example considered by Gauss, the five roots are $2,3,4,5,6$.
- For the first root, Gauss also uses the term $191.r$ (I believe $r$ is a abbreviation for "reminder").

Goettingen State and University Library, Cod. Ms. Gauß Schedae 5, p. 14.