Finite means "not infinite." Or, "In one-to-one correspondence with a natural number" (or, "in one-to-one correspondence with the set $\{1,2,\dotsc,n\}$ for some $n \in \mathbb{N}$", if you prefer.) You probably know what finite means.
The point of that quote is that, when we want to generalize our intuition about finite subsets of $\mathbb{R}^n$, there are two different properties of finite sets we can seize on to generalize.
One is that finite sets are discrete: i.e. each of the points is separated from all the others (about each point $v$ of your set, you can find a ball such that the only one of your points in the ball is $v$.)
Another is that the set is compact: this means that every open cover (every collection of open sets whose union contains all of your set) has a finite subcover. (This is obvious for finite sets. For each point $v$ in your set, just choose one set from the open cover that contains $v$.)
The only sets which are both discrete and compact are finite.
So, these two properties take different aspects of finite sets, giving you two big classes of subsets of $\mathbb{R}^n$ which are "like finite sets" in some way—but in two different ways. It's accurate to say sets of either kind are "like finite sets," but this can mean more than one thing.
When people say "compact sets are like finite sets," they don't mean that they're discrete: the points of a compact set aren't isolated from eachother. Instead, they mean that every open cover has a finite subcover. This gives a lot of well-behaved properties, similar to those enjoyed by finite sets:
- every function defined on a compact set achieves its maximum and minimum values, just like a function defined on a finite set
- if a sequence of points in the set has a limit, then the limit is also in the set (just like finite sets)
- the set must be bounded, just like finite sets
- the set must be closed, just like finite sets