The issue is one of semantics and terminology, not mathematics or economics.
When we talk about concepts like "continuous compounding" or even interest rates in general, it is important to be precise about what we mean, because as is so often the case in mathematics, formulas and equations can disambiguate what is not always clearly expressed through language.
For instance, when we think about interest earned on a fund, it is often convenient to conceptualize the amount earned over some fixed period of time, usually a calendar year. So for instance, if I suppose I invest $1000$ that earns $5\%$ interest per year, then at the end of one calendar year I would have $1050$. And if I reinvest this amount at the same rate, then at the end of the second year I would have $1102.50$, and so forth. This is the familiar "compound interest" that we have seen before:
$$A(n) = A(0) (1 + i)^n,$$ where $A(0)$ is the original amount invested, $i$ is the annual rate of interest, and $n$ is the whole number of years the fund is invested.
The more precise terminology for the value $i$, however, is the effective annual rate. This is the amount, irrespective of how the fund might be compounded during a year, by which the fund effectively grows.
But suppose we want to check on the value of our fund at the end of each calendar month, rather than each year. The bank, as you put it, isn't giving us more or less money; the rate of return on our investment is the same whether or not we look at the balance at some intermediate point in time. If we assume that the interest rate is not fluctuating over time, how much can we expect to see at the end of each month? That is to say, what is the effective monthly rate that is equivalent to the effective annual rate? This requires us to find the solution to the equation
$$(1 + j)^{12} = 1 + i,$$ where $j$ is the desired effective monthly rate. We get $$j = (1+i)^{1/12} - 1,$$ and for $i = 0.05$, this gives $j \approx 0.00407412$ or about $0.407\%$ per month. So at the end of the first month, you would see a balance of $1004.07$ in the account; at the end of the second, you would have $1008.16$, and so forth, until at the end of the twelfth month you would again have $1050$ exactly.
And we can extend this reasoning for finer time increments; e.g., if we want to check on the balance daily, the effective daily interest rate (assuming 365 days in a year) is about $0.0133681\%$. Again, we are assuming that interest is compounded at a constant rate (whereas in practice, interest rates fluctuate with market conditions, but this is more a topic for economics as opposed to mathematics).
However, effective rates for extremely short time periods are not convenient to report. As you can see, their value tends to $0$ as the time increment also tends to $0$ (as an exercise, what is the effective hourly rate, assuming 365 days in a year and 24 hours in a day?). This is where nominal rates of interest come into play. A nominal rate is defined to be the effective rate multiplied by the number of compounding periods over which the rate is converted; e.g., the nominal monthly rate that is equivalent to an effective annual rate of $5\%$ is $12(0.00407412) \approx 0.0488895$ or about $4.89\%$. We use the symbol $i^{(12)}$ to describe this rate; therefore, $$\left(1 + \frac{i^{(12)}}{12}\right)^{12} = 1 + i$$ and in general, the nominal $m^{th}$-ly rate that is equivalent to an effective annual rate of $i$ is given by $$i^{(m)} = m\left((1 + i)^{1/m} - 1\right).$$ (Note that the use of parentheses in the superscript is so that we do not confuse it for an exponent.) So in the daily case, $i^{(365)} \approx 0.0487934$ and in the hourly case, $i^{(8760)} \approx 0.04879030$. You can see that as the compounding frequency increases, the nominal rate tends towards a nonzero fixed number. In the limit, we get $$i^{(\infty)} = \log(1 + i) \approx 0.048790164169432.$$ This is what we call the force of interest and instead of $i^{(\infty)}$ we typically write $\delta$.
Again, every single rate, whether nominal or effective, monthly or daily or hourly, I have calculated so far in our discussion, is equivalent to the same effective annual rate of $5\%$, in the sense that at the end of the first year, the balance is still $1050$ on an initial investment of $1000$.
Now that we have properly described our notation and terminology, where does $e$ come from? Well, up to this point we have been characterizing the same effective interest rate in different ways by considering the equivalent nominal rate or the equivalent rate over different time periods. In this way, we still end up with the same return--the bank is not giving you any more or less money.
However, what if instead we consider the same numerical value for the nominal interest rate, but compound it over different periods of time? That is to say, for a given fixed nominal rate, say $5\%$, what is the equivalent effective annual interest rate as the conversion/compounding occurs more frequently?
In the case where $i^{(12)} = 0.05$, so we have a nominal monthly rate of $5\%$, the effective annual rate is $$i = \left(1 + \frac{i^{(12)}}{12}\right)^{12} - 1 = \left(1 + \frac{0.05}{12}\right)^{12} - 1 \approx 0.0511619,$$ or about $5.12\%$. If the nominal daily rate is $i^{(365)} = 5\%$, then the effective annual rate is $$i = \left(1 + \frac{i^{(365)}}{365}\right)^{365} - 1 \approx 0.0512675,$$ and so forth. Unlike our earlier discussion, each of these situations results in a different accumulated value of the fund at the end of the year because the effective rate of interest is different: the more frequently the same nominal rate of interest is converted/compounded, the more money is earned. With annual compounding, the amount is $1050$; with monthly compounding, you would have $1051.16$; with daily compounding, you would have $1051.27$. But as you can see, as the compounding frequency increases without bound, there is a limit to the amount of money you will ultimately earn in a year, just like in our earlier discussion about the force of interest: If you compound infinitely frequently, you only get about $1051.271096$. What is this value in relation to $5\%$ and $1000$? It is $$1000 e^{0.05}.$$ This is because in this case, the force of interest is $\delta = 0.05$.
So to emphasize, in the first part of our discussion, the effective rate and hence the annual return is fixed, while the nominal rate is allowed to vary with the compounding frequency, until we arrive at some equivalent force of interest. In the second part of the discussion, the nominal rate and hence the force of interest is fixed, while the effective rate and annual return are allowed to vary.