You are right: There should be a difference between $A \nvdash \bot$ and $\vdash \neg (A \to \bot)$, and there is.
As you note, $A \to \bot$ is logically equivalent to/taken to be the meaning of the abbreviation $\neg A$, so $\neg(A \to \bot)$ is equivalent to $\neg \neg A$, and this is in turn equivalent to (and from which is derivable) $A$. That is, $\vdash \neg(A \to \bot)$ entails (assuming semantic completeness of the system) $\vdash A$.
$A \vdash \bot$ would mean that a contradiction can be derived from $A$; assuming soundness that means that $A$ is contradictory, i.e. false under all interpretations. $A \nvdash \bot$ says that this is not the case and thus means that $A$ is satisfiable, i.e. not false under all interpretations, i.e. true under at least one interpretation.
If $A$ were true under all interpretations, it would be tautological, and assuming completeness, it should be derivable: $\vdash A$, and with the above equivalene, $\vdash \neg(A \to \bot)$.
The quantification over interpretations is important here; it does not make sense to say that A is "true" without specifying under which interpretation, becauase truth is only defined relative to interpretations (interpretation = valuation functions in the case of propositional logic, and structures with domain and interpretation function in predicate logic).
However, $A$ being satisfiable (= "not always false") does not imply that $A$ is tautological (= "always true"): It may be true under some but not all interpretations. Hence, again taking the proof system to be sound, from the non-derivability of a contradiction from $A$ we can not infer derivability of the negation of $A \to \bot$ (which would entail derivability of $A$):
$A \nvdash \bot \ \nRightarrow \ \vdash \neg(A \to \bot)$, although the two don't exclude each other ($A$ may be derivable, in which case also no contradiction can be derived from it).
If both $A \nvdash \bot$ (i.e. $\nvdash \neg A$) and $\nvdash \neg(A \to \bot)$ (i.e. $\nvdash A$), then we do indeed have the case that $A$ is undecidable or independent of the theory, and the proof system is syntactically incomplete.