The simplest case is when the group is equipped with a topology with respect to which it is compact (Hausdorff), so the proof of Maschke's theorem still works and the Peter-Weyl theorem is available. In particular, the representation theory of compact Lie groups is very well understood. The representation theory of noncompact Lie groups is still a major area of modern study, with ties to algebraic geometry, harmonic analysis and number theory. The representation theory of profinite groups such as Galois groups is also a major area.
The representation theory of infinite discrete groups is, as far as I know, extremely hard in general. Some work has been done on representations into $\text{SL}_2$ and related groups; see character variety and the discussion and references here, for example.
For some indication of how hard these questions are, the representation theory of $F_2$ is essentially a mystery: that is, no one knows a reasonable way to classify pairs of matrices up to simultaneous conjugation.