Based on NIST SP 800-38D section 5.2.1.1, it seems that the maximum length of plaintext is 2^39-256 bits ~ 64 GB.
We've got 100+GB files in genomics that need to be GCM encrypted so are concerned about hitting this.
So two questions:
The source of the limitation lies in the fact that GCM has a fixed block counter using a 32-bit integer. Since the block size is $2^7$ bits, the total amount that can be encrypted with the CTR component is $2^{39}$ bits.
The first limit reducing this by 128-bits is the fact that the block counter starts at 1 and not 0, at least with a 96-bit nonce. Nonce sizes other than 96-bits are know to have reduced security. The second limit reducing this by 128-bits is because the CTR component is also used to encrypt the final GHASH prior to tag output.
Implementation wise, you should not be allowed to cross this limit. In theory, the security breaks down. There are different security limits for the amount of data encrypted, and for the amount of tags generated. With a long message, an $n$-bit tag provides $n-k$ bits of authentication security for a $2^k$ block message, in your case this brings it down to around $2^{63}$ with a file between 100 and 128 GB when used with a 96-bit tag, much lower than the $2^{128}$ one may expect. The worst case scenario is recovery of the hash subkey $H$, allowing forgery of tags with the same key.
Encrypting a full size 128-bit tag with a separate key in ECB mode should eliminate that problem, as the bits will no longer be malleable. That can be done as a wrapper around a GCM implementation with minimal modification and almost 0 overhead. The implementation would need to be modified to use a larger block counter, and the nonces would need to be shorter. You would no longer be able to call it GCM, but would be able to make use of all the features and code that make GCM implementations fast and timing attack resistant. A 32-bit nonce and a 40-bit block counter (with no additional GHASH) would allow almost 16TB per nonce, without the restriction of changing keys for every message if you encrypt the tag with a second key. If you even consider something like this, a professional consult on both the modifications and the code to do it would be highly recommended, new security proofs would need to be developed.
Other solutions to the problem including not using GCM, but a different mode like Poly1305-AES. OCB mode also has a suggested limit of 64GB per key, CWC mode has the same limit as GCM.
I do not agree with the other answers and comments given here. The use of the 96-bit nonce gives the best bounds, but it is certainly not the only way to use GCM. Also, the degradation is gradual. It is not the case that anything else is insecure. Having said this, it is completely insecure to encrypt beyond $2^{32}$ blocks using a 96-bit counter since the same counter will repeat, and this is a complete disaster.
The easiest solution to this is to use an 88-bit nonce. You can then encrypt up to $2^{47}$ bits which is enough.
If there are machines that really are worth over 200 million dollars involved, you presumably have a secure physical environment. That means there is another option: Do all cryptography OUTSIDE of the expensive, inflexible machines, and transfer the data to/from the machines in plaintext, using physically secure networks, air-gapped from the outside world. Transferring data over a network in plaintext is not a vulnerability if the attacker cannot gain access to the network to tamper with it (because it is in a locked room witb an armed guard).
