The "hard part" about GCM implementation is resistance to side-channel attacks, especially cached-based. GCM is the combination of AES-CTR, and a custom MAC that relies on multiplications in a binary field (GF(2128)). Efficient implementation of that operation classically uses tables, which can lead to cache-timing attacks because the accessed table slots will depend on the secret data (though cache-timing attacks have never, to my knowledge, been spotted "in the wild", they work very well in lab conditions so the common wisdom is that they are a plausible threat).
If you have a recent enough x86 CPU then you have the AES-NI opcodes, which help for AES but also for multiplications in binary fields, making GCM both resistant to side-channel leaks and very efficient. But smaller devices do not necessarily have the luxury of such a hardware-assisted implementation. In fact, it may be argued that performance is much more likely to be an issue on small embedded CPU (say, the low-end 32-bit ARM CPU) than on big PC, where encryption only uses a small fraction of the available computing power.
For small CPU, Poly1305/Chacha20 will be faster, with a smaller footprint, and more resistance to side-channel attacks. Techniques for making a constant-time implementation of AES are still applicable (like the Kasper-Schwabe implementation) but will offer lower performance because a 32-bit small CPU has only 32-bit registers (the Kasper-Schwabe code uses bitslicing techniques to run 8 parallel AES instances over 128-bit registers, but with 32-bit registers this goes down to 2 parallel AES instances). For the binary field part, one can use plain integer multiplications, but with most bits cleared to avoid problems with carries, so there will be a lot more multiplications than with a prime-field based MAC like Poly1305.
GCM is there to stay because it got NIST blessing and thus made its way to many standards; it even convinced CPU vendors to include specialized opcodes, and this in turn will reinforce its position.
Meanwhile, research on AEAD modes is still active, and there is an ongoing competition with interesting candidates (e.g. NORX). Thus, while there is a bit of dissatisfaction over GCM because of implementation issues, the succession is still open. Poly1305+ChaCha20 got a little headstart by being there first, but that does not make it an automatic fallback.
A common problem to both GCM and Poly1305+ChaCha20 is the need for a non-repeating IV. Such an IV is easily managed in the context of a streamed protocol (such as SSL/TLS) but it can be a source of trouble for encryption of independent messages -- IV reuse is a mortal sin and is not easily tested for in a given implementation. An IV-less encryption system would need to be a two-pass method (which is not necessarily a problem for encrypting messages); a possible candidate would be a counter-intuitive MAC-then-encrypt construction, with (say) HMAC computed over the plaintext, and the HMAC value being used as IV for AES-CTR encryption.
