An element that belongs to no circuit is called a coloop. Equivalently, an element is a coloop if it belongs to every basis.
Circuits are minimal dependent sets, while bases are maximal independent sets.
Now let $c\in E$ be a coloop, so it belongs to no circuit. Let $B\subseteq E$ be a basis and assume $c\notin B$. Since $B$ is maximal independent, $B\cup\{c\}$ is dependent, so it contains a circuit $C\subseteq B\cup\{c\}$. Now $C\subsetneq B$, since $C$ would be independent otherwise. So we conclude $c\in C$, which contradicts $c$ being a coloop. Thus, our assumption was wrong and we have $c\in B$, so $c$ is an element of every basis.
I leave it to you to find a proof for the other implication.
From the equivalence, I suspect that the union of all circuits and the intersection of all bases do not overlap? Is it correct?
Indeed, the union of all circuits does not intersect the intersection of all bases, since the latter is the set of coloops, while the former doesn't contain any coloop.
I also saw that a coloop in a matroid can be defined as a loop in its dual matroid. Why is this equivalent to the previous two definitions?
For the dual matroid, the bases are exactly the complements of the original bases. So something that is a loop in the dual matroid belongs to the complement of no basis of the matroid itself, thus is contained in every basis, so it is a coloop.
The complement of a circuit in a matroid is a hyperplane in its dual matroid. In its dual matroid, the complement of the hyperplane is a coloop. So in a matroid, is the complement of a hyperplane inside every basis?
A hyperplane is a flat of rank $r-1$, where $r$ is the rank of the matroid. Thus, adding any element to a hyperplane gives you a set of full rank, so it contains a basis. Note that a hyperplane doesn't always have $|E|-1$ elements, so your second statement can't be true in general, since the coloops are elements.