You can repeat the first capture group (?1) instead of repeating the whole pattern with (?R) and capture what is inside the {} with group 2
\\textbf({([^{}]*+(?:(?1)[^{}]*)*+)})
\\textbf Match \textbf
( Capture group 1
{ Match a { char
( Capture group 2
[^{}]*+ Optionally match any char except { } with a possessive quantifier
(?: Non capture group to match as a whole
(?1)[^{}]* Recurse the first subroutine and optionally match any char except curly's
)*+ Close the non capture group and optionally repeat using a possessive quantifier
) Close group 2
} Match a } char
) Close group 1
Regex demo
Note that if you use re.findall, you will get all values of the capture groups returned, and this pattern has 2 capture groups.
You can use re.finditer instead and get the group 2 value:
import regex
pattern = r"\\textbf({([^{}]*+(?:(?1)[^{}]*)*+)})"
cnt = ("\\textbf{adadasas}\n"
"\\textbf{adadasas \\textit{xxx} adasda {xxx}}\n"
"\\textbf{adadasas {} {} {} dxxxx}\n"
"{adadasas {} {} {} dxxxx}")
matches = regex.finditer(pattern, cnt)
for _, match in enumerate(matches, start=1):
print(match.group(2))
Output
adadasas
adadasas \textit{xxx} adasda {xxx}
adadasas {} {} {} dxxxx