In other words, is the "provability of provability" a valid notion? (Moreover, what about the "provability of provability of provability of, ..."?)
Yes, "provability of provability" makes total sense: as you indicate, we do this relative to some kind of formal system with inference rules: using the rules, the system can create formal proofs and thus "prove" statements.
However, one question that quickly comes up is: can that formal system provide a formal proof of all the things we would like it to prove? For example, let's consider a simple formal system that we want to use for propositional logic. We know that $P \to P$ is a logical tautology, so we would like our formal system to be able to formally proof that. Indeed, can the system prove all logical truths from propositional logic?
Well, here we can do some meta-logic: we consider the nature of the rules, and try to come with a meta-logical proof (which is not a formal proof, but more of a mathematical proof) that the formal system can indeed prove all propositional logical truths. Or: we may find (and mathematically prove!) that the formal system is unable to prove all logical truths from propositional logic.
So, yes, we may be able (mathematically) prove that certain things are provable in some system. Or we may be able to (mathematically) prove that other things are not be provable in some system. If we can mathematically prove that some formal system is able to formally prove all the things we want it to be able to prove, we say that the formal system is complete.
Now, what about "(un)provability of (un)provability of (un)provability"? Does that make sense? Sure! We could, for example, come up with a formal system that formalizes the very meta-logical reasoning we did for the earlier proof of (un)provability. As such, we would have a formal system for doing meta-logic.
Working within that system, we can come up with formal proofs of formal (un)provability. But stepping outside of that system, we can try and prove things about the abilities of that system, i.e. we'd get mathematical proofs of the formal (un)provability of formal (un)provability. I suppose you could call that doing meta-meta-logic, but I believe it's still considered just plain meta-logic.
Here, by the way, is something fun to read about how you can stack these (un)provability claims.
OK, but none of this answers your first question:
Is there a proposition that cannot be proven to be provable or unprovable within a formal system?
Now, this question is a little ambiguous.
First, I am going to assume that you are asking about the mathetical (i.e. non-formal) unprovability of the claim that some statement is (un)provable within some formal system, rather than that you are considering some 'meta' formal system that is unable to prove the (un)provability of some statement ... given the rest of your post, it is unlikely that you even considered that second interpretation.
Another ambiguity is this: Do you have some specific formal system in mind, or is this supposed to generalize over all formal systems? That is: Are you looking for a proposition $P$ for which we cannot mathematically prove that $P$ is provable within some specific formal system (call it $S_0$) as well as for which we cannot mathematically prove that $P$ is unprovable within $S_0$? Or are you looking for a proposition $P$ such that given any formal system $S$, we cannot mathematically prove that $S$ cannot prove $P$ as well as for which we cannot mathematically prove that $S$ cannot prove $P$?
Now, I assume you mean the latter, although I will also assume that you want to restrict the set of formal systems to those that uses the very formal language tat was used to construct the formal proposition $P$.
OK, so with those assumptions, the answer is no. For example, consider some formal system that has as one of its formal inference rules the rule of Hokus Ponens:
Hokus Ponens
$\therefore \varphi$
(that is, this rule allows you to write down any statement of your language at any point in the proof!)
Now, this is of course a silly rule. It is in fact a logically invalid rule. But ... one can create a formal system for which one simply stipulates that this is one of its inference rules! OK, so note with such a rule, any proposition (from the language being used) is formally provable! In fact, the mathematical proof of the formal provability of any statement $P$ in the system is completely trivial! So no, there is no statement $P$ such that no matter what formal system $S$ we use, we cannot mathematically prove the provability nor the unprovability of that proposition within that formal system. Or, to put things differently: if you have some proposition, and want to formally prove it using some formal system, then you can do this simply by picking this 'hyper-complete' formal system!
OK, you say, but that's cheating! You are using a system that can prove anything, including statements that aren't 'true' at all. For example, this system could prove the statement $P$ (which is not a logical truth) and even the statement $P \land \neg P$ (which is the opposite of a logical truth: a logical contradiction!). So, how about we restrict ourselves to systems that can only prove things that are actually true? Or maybe better put: a system that only makes logically valid inferences? Note that we call such systems sound.
Well, how about this: consider a system that has no inference rules at all! Such a system is clearly sound: it is unable to prove anything that doesn't follow for the simple reason that it cannot prove anything at all! But note, given any statement $P$, we can (again, trivially!) mathematically prove that this system is unable to prove that proposition ... and hence it is again not the case that there is a statement $P$ such that no matter what formal system $S$ we use, we cannot mathematically prove the provability nor the unprovability of that proposition within that formal system.
Hmmm, ok, so how about we restrict ourselves to formal systems whose rules are both logically sound and logically complete? We certainly have such systems for both propositional logic and first-order logic. Indeed, this is really just the normal situation.
Well, now Peter's comment becomes relevant: We know (hanks to Turing) that first-order logic is undecidable, meaning that there is no one systematic method that can tell for any statement whether it follows from some other statements ..i.e. whether it is provable or not. In other words, given any formal system (of sufficient strength), we know (i.e. it is mathematically provable) that there are statements that that system cannot prove.
OK, but that is not quite the same as what you asked (I think ... there is in fact another ambiguity here): I believe you were asking whether there are statements for which we cannot mathematically prove that no matter what formal system we use, whether that system can prove that statement or not.... and the answer is still no, even if the system is logically sound and complete.
Again, we can do something silly: Suppose you give me a statement $P$. Then I can simply create a formal system that has that very statement $P$ as one of its axioms... in fact we could make it its only axiom. Well, then clearly that system will be able to prove that statement. And again, the mathematical proof of that face is trivial! So, it is mathematically provable that that statement is provable by some formal system ... meaning that no, there is not a statement $P$ such that for any system $S$ it is not mathematically provable that $S$ can prove $P$.
Indeed, this trick shows that any statement is formally provable ... by some formal system.
Of course, whether the provability of that statement means that it is 'true' is a whole other matter...
