Q. In what sense do homology and homotopy groups "count" or "detect" holes? And when do they differ in their hole counting?
I'm seeking a high-level view of these questions, although perhaps they cannot be answered without delving into technical details. A good reference that directly addresses these questions would also be appreciated.
Basically both of these objects count holes in a space by making precise different concepts of what a hole is.
The fundamental idea underlying both definitions is that a hole in a space is detectable by (continuously) mapping a closed $n$-dimensional surface into the space and checking whether or not that surface can be filled in by an $(n+1)$-dimensional object whose boundary is the $n$-dimensional surface inside the space. If the surface can't be filled in, then we can think of it as surrounding a hole in the space. For example, the fundamental group, $\pi_1(X)$, is built by mapping circles into $X$ and checking if they can be filled in by a disk.
I should point out now that not being able to fill in our surface just means that we surround at least one hole, but we don't necessarily know how many it surrounds. In fact, it could surround the same hole more than once. Consider a closed path in $\mathbb{R}^2-\{(0,0)\}$ that goes around the origin twice before joining up with itself again.
Of course if we have two surfaces that can't be filled in inside our space, they might represent the same hole in the space, so we need an equivalence relation on our maps from surfaces into the space that tells us whether or not we consider these maps to represent the same hole in the space.
The difference between $\pi_n$ and $H_n$ is in how exactly they make precise the idea of mapping a closed $n$-dimensional surface into a space and the equivalence relation between maps of surfaces into the space.
Homotopy Groups
For $\pi_n$ we restrict the test surfaces that we use to look for holes. This makes things simpler in some ways and more complicated in other. For $\pi_n$ we only consider continuous maps from the $n$-sphere to $X$. Filling in a map $f:S^n\to X$ corresponds to giving a map $F:D^{n+1}\to X$ which restricts to our original map $f$ on $\partial D^{n+1}=S^n$. We also need an equivalence relation that determines when two maps $f,g:S^n\to X$ represent the same hole (or set of holes) in $X$. Well, if we can find a continuous family of maps $H:S^n\times [0,1]\to X$ such that $H(-,0)=f(-)$ and $H(-,1)=g(-)$, then $f$ and $g$ must surround the same hole (or set of holes) in the space. This relationship is called homotopy, and if we can find such an $H$, then we say $f$ and $g$ are homotopic.
However, I've left something important out of the definition of $\pi_n$ here. While this is conceptually correct and it should describe the same set as the usual definition of $\pi_n$ correctly when $X$ is connected, this isn't actually the definition used in practice. We have to make a small modification. Instead of letting our maps be arbitrary maps $S^n\to X$ we fix a basepoint $x\in X$ and a basepoint $*$ in $S^n$ and require that our maps $f:S^n\to X$ send $*$ to $x$, and that our homotopies $H$ have the property that $H(*,t)=x$ for all $t\in [0,1]$. This base-pointed version is written $\pi_n(X,x)$. When $X$ is path-connected, it turns out not to matter what choice of $x$ was made (up to isomorphism), so if you see $\pi_n(X)$, that notation does still refer to the base-pointed version, but either the base point is implicitly specified elsewhere, or $X$ is path-connected so we can take any basepoint we please.
The reason we want to do this is because when we fix a basepoint for our maps we can actually define a notion of composition of surfaces that makes $\pi_n$ into a group. The idea is the following. Imagine collapsing the equator of an $n$-sphere into a point. The resulting space consists of 2 $n$-spheres that meet at a single point (this is called the wedge sum of the 2 $n$-spheres, written $S^n\vee S^n$). Now if we're given two maps $f,g:(S^n,*)\to (X,x)$ that preserve base points, then we can define a map $f\vee g : (S^n\vee S^n,*) \to (X,x)$ by taking the map to be $f$ on the first copy of $S^n$ and $g$ on the second copy of $S^n$, which makes sense, since the two copies of $S^n$ only intersect at a single point, the base point, and $f$ and $g$ both second the basepoint to $x$ in $X$. Then we can just compose this map $f\vee g$ with the "collapse the equator" map $S^n\to S^n\vee S^n$ to get a new map $f+g : S^n \to S^n\vee S^n \xrightarrow{f\vee g} X$.
This gives a group structure on $\pi_n$ for $n\ge 1$, and this group structure is commutative for $n\ge 2$. (When $n=1$ the operation is usually written multiplicatively as $f*g$ or $f\cdot g$ rather than $f+g$ so as to avoid suggesting that it is commutative.)
What does this group operation represent in terms of holes in the space $X$? Well, as I mentioned before, a surface might surround a set of holes in the space, and it might surround them multiple times. The group operation basically combines the sets of holes that each surface surrounds and adds them together. This might sound like it doesn't give a group operation, since it doesn't sound like there are inverses from this description, however spheres are orientable surfaces, and essentially we end up with a situation where surrounding a hole in one orientation can be cancelled out by surrounding the hole with the opposite orientation.
Again, it's helpful to think of the case of $\pi_1(\mathbb{R}^2-\{(0,0)\}, (1,0))$. If we take a path that goes around $(0,0)$ counterclockwise and then follow it up with a path that goes around $(0,0)$ clockwise, then we can homotopy that composite path to the constant one by simply pulling the loop back along the circle, so the composite ends up being homotopic to the trivial path that just stays at the basepoint and doesn't go anywhere.
This is also helpful in understanding conceptually why $\pi_1$ is not abelian but $\pi_n$ for $n\ge 2$ is. From the perspective of a path, there is a meaningful notion of order, so we can think of our map from $S^1$ to $X$ as going around holes in a particular order, and we may have to unwind our path by traversing our path backwards according to that order. However for a map from $S^2$ or higher in there's no longer a meaningful notion of order, so there is no real order our surfaces go around the holes in $X$.
(Singular) Homology Groups
For $H_n$ we make things a little more complicated initially by allowing more test surfaces than just spheres, but it ends up making things simpler overall.
For singular homology we actually start by taking maps of simplices $\Delta^n\to X$. The $n$-simplex $\Delta^n$ is essentially the $n$-dimensional analog of a triangle. For example, in small dimensions, the 0-simplex is a point, the 1-simplex is a line segment, the 2-simplex is a triangle, and the 3-simplex is a tetrahedron. A useful property of simplices is that their boundaries are also made up of simplices, namely the boundary of the $n$-simplex is made up of $n+1$ simplices of dimension $n-1$. The intuitive description of the use of this is that if we build a space up out of simplices (these are called simplicial complexes) by gluing them together along their faces the resulting space will still have a well-defined notion of boundary, and this will also be a simplicial complex. (This is intuitively correct, but not precise.)
Basically the idea then is that $H_n(X)$ is defined by taking maps from $n$-dimensional simplicial complexes whose boundary is empty to $X$. (The boundary of a simplicial complex being empty means that the simplicial complex forms a closed $n$-dimensional surface essentially.) The equivalence relation we impose here is that two maps from potentially two different $n$-dimensional simplicial complexes $A$ and $B$ to $X$ are homologous if there is an $n+1$ dimensional simplicial complex $W$ whose boundary is $A-B$ and a map from $W$ to $X$ restricting to the correct maps on $A$ and $B$. Note the minus sign. The key problem I've been glossing over here is that everything here needs to be oriented. Our simplices need to be oriented and the gluings in our simplicial complex need to respect the induced orientations on the faces of the simplices. With the orientation properly handled, taking the boundary is an oriented operation, so $-B$ makes sense. It's the simplicial complex $B$ with the opposite orientation.
Now even oriented this still isn't precisely the definition of singular homology, but this description is more useful from a high level point of view and for making sense of the way in which homology measures holes in a space.
As in the homotopy group case this definition makes $H_n$ a group. Basically addition is given by taking disjoint unions of simplicial complexes and inverses are given by reversing the orientation of a simplicial complex. Note that unlike in the homotopy group case we don't need to choose a basepoint in $X$, and this is basically because we are much less restrictive in what closed $n$-dimensional surfaces we allow. Again the addition basically corresponds to taking oriented sums of the sets of holes that a surface encompasses.
In this case, however, the sum is always commutative, since the disjoint union is commutative (the sum also makes sense for $H_0$, whereas this was not true for $\pi_0$).
Relationship between Homotopy and Homology Groups
Since $\pi_n$ detects holes by using the surface $S^n$, and $H_n$ detects holes by using basically any $n$-dimensional closed surface, you might think there should be a relationship between the two objects. And you'd be right to think so. There is always a map $\pi_n(X,x)\to H_n(X)$ for any choice of basepoint $x$, called the Hurewicz homomorphism, which essentially just says well if we have map $f:S^n\to X$, then we can take a simplicial complex homeomorphic to $S^n$ and regard the map $S^n\to X$ as an element of $H_n(X)$. There are some details to be checked, of course, namely that homotopic maps get sent to the same homology class, and also that this is a group homomorphism.
In fact, the relationship is even better than this. There is a theorem due to Hurewicz that says that if $X$ is path-connected, and $n$ is the first integer such that $\pi_n(X)\ne 0$, then if $n\ge 2$, the Hurewicz homomorphism $\pi_n(X)\to H_n(X)$ is an isomorphism, and if $n=1$, the map $\pi_1(X)\to H_1(X)$ induces an isomorphism between the abelianization of $\pi_1(X)$ and $H_1(X)$.
In other words, in the first dimension that the space $X$ has holes, $\pi_n$ and $H_n$ agree about what kind of holes $X$ has.