I'm hesitating to choose my definition for "p-subgroup".
If i only consider finite cases, i would define it as a group with the order which is a power of $p$.
If i should consider infinite cases too, i would define it as a group such that every order of an element is a power of $p$.
Is there a case in which the concept of infinite p-group is useful?
Sure. An infinite dimensional vector space over a field with $p$ elements is probably the simplest example. But there are some really wacky groups called "Tarski Monsters" that provide examples of infinite simple groups with every proper, non-trivial subgroup of order $p$.
Infinite Sylow Theory
In finite group theory it is common to define a $p$-group to be one of order a power of $p$, but at the same time Cauchy's theorem says this is equivalent to all elements having order a power of $p$. So in infinite group theory, this becomes the definition of a $p$-group. While in finite group theory a Sylow subgroup is usually one of maximal prime power order, a slight tweaking gives the equivalent condition of being a maximal $p$-subgroup (not a subgroup which is maximal and a $p$-group, but rather a subgroup which is maximal among $p$-subgroups). One easily checks that the property of being a $p$-subgroup is preserved by taking unions of chains of them, so by Zorn's Lemma every infinite group has a Sylow $p$-subgroup. The Sylows may all be trivial, though, as they collectively miss a certain type of element: the torsionfree elements.
So we've already extended Sylow I to the infinite context, what about II and III? I leave it as an exercise to check that II and III both extend for a group $G$ and prime $p$ assuming that $G$ has only finitely many $p$-Sylows. There are groups for which there aren't finitely many, and II/III fail.
One can find more infinite Sylow theory in the papers of Baer and Neumann.
Tarski Monster Counterexamples
Many intuitive claims in group theory are (or have been, or can be) disproven by Tarski Monsters.
Burnside's problem. Clearly there are finitely-generated groups which are infinite (the infinite cyclic group, for instance, or other free groups) and there are torsion (aka periodic) groups which are infinite (e.g. the additive quotient group $\Bbb Q/\Bbb Z$ or an infinite-dimensional vector space over a finite field). It would seem obvious that a fininitely-generated torsion group must be finite, right?
There are only finitely many letters (taken from a generating set) to write words out of, and not only is each letter but every single word is torsion! And yet, nope. Obviously our problem must be that we didn't bound the torsion, thus yielding words of potentially arbitrary length: if we stipulate our group must have finite exponent, what then? Still, no luck. Well, what if finite isn't enough, we just have to restrict the exponent even further - what if we stipulate exponent is as small as it can go without being trivial: a prime $p$? Still the answer is no! This means every element has order $p$, but what if we go even further and stipulate every proper subgroup has order $p$? Nope!
Pro-$p$ Groups
In this day and age, with the technology of topological groups, why settle for a topologyless notion of $p$-groups? While infinite group theory has us generating new $p$-groups through unions (adding in more and more elements of $p$-power order), profinite group theory has us proceed in a dual manner by defining a pro-$p$ group to be an inverse limit of finite $p$-groups. Compare:
Typical examples include: $p$-adic integers $\Bbb Z_p$, matrix groups over $\Bbb Z_p$, discrete finite $p$-groups.
The analysis of pro-$p$ groups has been used, alongside $p$-adic integration, zeta functions of groups and modular Lie algebras, to study subgroup growth in general profinite groups and classify the familiar finite $p$-groups not by order or nilpotence class but coclass, their difference. Specifically, the coclass theorems state there are only finitely many pro-$p$ groups of any given coclass.
Another way both to form infinite examples where Sylows theorems can fail, or behave as expected, is to use amalgams (or free products with amalgamation). For example, if we take the free product of two cyclic groups of prime order $p,$ we get an infinite group in which any finite subgroup has order dividing $p,$ but the two generating subgroups of order $p$ are not conjugate. The infinite dihedral group $\langle t,u,: t^{2} = u^{2} = 1 \rangle$ is the "easiest" example, with $p=2.$ On the other hand, if $A,B$ are (possibly infinite) groups which each have a unique conjugacy class of maximal finite $p$-subgroups, and have isomorphic maximal finite $p$-subgroups, say $P$, then $A*_{H}B$ still has a unique conjugacy class of maximal finite $p$-subgroups, where $H$ is any common subgroup of $A$ and $B$ containing an isomorphic copy of $P.$
I suppose that for infinite groups, you have to be careful by what you might mean by a Sylow $p$-subgroup. One definition might be a maximal $p$-subgroup, which contains every other $p$-subgroup up to conjugacy. Another definition might be a maximal finite $p$-subgroup, which contains every other finite $p$-subgroup up to conjugacy. It is possible to have infinite groups which have Sylow $p$-subgroups in the second sense, but not in the first. There may be easier ways to do this, but you would (for sufficiently large $p$), take two Tarski monsters $S$ and $T$ such that all subgroups of order $p$ are conjugate in $S$, and likewise in $T$ (these exist)- it might be that $S$ and $T$ are isomorphic, I don't care. For the amalgam $A = S*_{C}T$, where $C$ is a common cyclic subgroup of order $p$ of $S$ and $T.$ Then all finite $p$-subgroups of $A$ are conjugate to subgroups of $C$, but $S$ and $T$ are maximal (infinite) $p$-subgroups of $A$ which are not conjugate in $A$- and furthermore, every infinite $p$-subgroup of $A$ is conjugate to either $S$ or $T.$
