When we add the extra day to the calendar every 4 years it's to "correct" the calendar under the assumption that the year's length is 365.25 days. Every year, you're .25 days off, so after 4 years, you're 1 day too far ahead, so add in an extra day to resync.
But in reality it's more like 365 days, 5 hours, 48 minutes, 45 seconds. In other words it's not 365.25 days exactly, we're over-stating it by 675 seconds.
Every 4 years when we add that day back we're actually adding 2700 seconds too much!
After doing this 32 times though (after 128 years), we'd have added 2700 * 32 = 86400 seconds too much. But this is exactly the number of seconds in a day, (24 * 60 * 60).
So why aren't we subtracting a day back out every 128 years? Instead we're subtracting the day out after 400 years instead.