I am just learning about the finite field theory for channel codes. My understanding is that when constructing a ${\rm GF}(p^m)$, where $p$ is a prime number and $m$ is a positive integer, one has to use polynomials as some of the elements. In contrast, if one is constructing a ${\rm GF}(p)$, where $p$ is a prime number, the set is simply $\{0,1,...,p-1\}$.
However, for a low order finite field, for example, ${\rm GF}(2^2)$, I don't understand why can't one just use $\{0,1,...,3\}$ with mod-4 addition and multiplication, but has to use $\{0,1,x,x+1\}$ with mod-$g(x)$, where $g(x)$ is the primitive polynomial of order 2.
Thank you.