The expression abs(x)**2 + 6*abs(x) + 5 does not actually contain abs(x) + 1 anywhere, so there is nothing to substitute for.
One can imagine changing it to abs(x)**2 + 5*(abs(x) + 1) + abs(x), with the substitution result being abs(x)**2 + 5*y + abs(x). Or maybe changing it to abs(x)**2 + 6*(abs(x) + 1) - 1, with the result being abs(x)**2 + 6*y - 1. There are other choices too. What should the result be?
There is no general approach to this task because it's not a well-defined task to begin with.
In contrast, the substitution f.subs(abs(x), y-1) is a clear instruction to replace all occurrences of abs(x) in the expression tree with y-1. It returns 6*y + (y - 1)**2 - 1.
The substitution above of abs(x) + 1 in abs(x)**2 + 6*abs(x) + 5 is a clear instruction too: to find exact occurrences of the expression abs(x) + 1 in the syntax tree of the expression abs(x)**2 + 6*abs(x) + 5, and replace those subtrees with the syntax tree of the expression abs(x) + 1. There is a caveat about heuristics though.
Aside: in addition to subs SymPy has a method .replace which supports wildcards, but I don't expect it to help here. In my experience, it is overeager to replace:
>>> a = Wild('a')
>>> b = Wild('b')
>>> f.replace(a*(abs(x) + 1) + b, a*y + b)
5*y/(Abs(x) + 1) + 6*y*Abs(x*y)/(Abs(x) + 1)**2 + (Abs(x*y)/(Abs(x) + 1))**(2*y/(Abs(x) + 1))
Eliminate a variable
There is no "eliminate" in SymPy. One can attempt to emulate it with solve by introducing another variable, e.g.,
fn = Symbol('fn')
solve([Eq(fn, f), Eq(abs(x) + 1, y)], [fn, x])
which attempts to solve for "fn" and "x", and therefore the solution for "fn" is an expression without x. If this works
In fact, it does not work with abs(); solving for something that sits inside an absolute value is not implemented in SymPy. Here is a workaround.
fn, ax = symbols('fn ax')
solve([Eq(fn, f.subs(abs(x), ax)), Eq(ax + 1, y)], [fn, ax])
This outputs [(y*(y + 4), y - 1)] where the first term is what you want; a solution for fn.