In GAP, AllHomomorphismClasses is (and I'm writing this as the person who wrote the function) a "toy" function that is provided to try small examples in class. (Concretely, Gallian's textbook has exercises about homomorphisms between dihedral groups.) It tries out all possibilities in a backtrack search and thus is likely to be unperformative for groups of order more than a few hundred that cannot be 2-generated.
What should work substantially better (for groups that are still reasonably small) is to test all subgroups and quotients for isomorphisms. This uses the fact that the isomorphism test performs typically much better than the brute-force image search.
The following function (which takes just a few seconds for the given group) does this:
EndoSeeds:=function(G)
local u,n,e,a,le,q,v,i,j,iso,fingerprint,asAuto,nat,qid;
fingerprint:=function(sub)
if Size(sub)<2000 and not Size(sub) in [512,1024] then
return IdGroup(sub);
else return Collected(List(ConjugacyClasses(sub),
x->[Order(Representative(x)),Size(x)]));
fi;
end;
a:=AutomorphismGroup(G);
u:=List(ConjugacyClassesSubgroups(G),Representative);
asAuto:=function(sub,hom) return Image(hom,sub);end;
n:=NormalSubgroups(G);
n:=List(Orbits(a,n,asAuto),x->x[1]);
e:=[];
# go through normal subgroups in deescending order
SortBy(n,Size);
n:=Reversed(n);
for i in n do
if Size(i)=1 then
q:=G;
le:=[IdentityMapping(G)];
else
nat:=NaturalHomomorphismByNormalSubgroup(G,i);
q:=Image(nat);
qid:=fingerprint(q);
v:=Filtered(u,x->Size(x)=Size(q) and fingerprint(x)=qid);
# fuse under automorphisms
if Length(v)>1 then
v:=List(Orbits(a,v,asAuto),x->x[1]);
fi;
le:=[];
for j in v do
iso:=IsomorphismGroups(q,j);
if iso<>fail then
Add(le,nat*iso);
fi;
od;
fi;
Print("Factor order ",Size(q),", ",Length(v)," subgroups, ",
Length(le)," maps\n");
Append(e,le);
od;
return e;
end;
It returns representatives of homomorphisms $\phi\colon G\to G$, up to the action of $\mbox{Aut}(G)\times\mbox{Aut}(G)$ on kernels and images, as well as the action of $\mbox{Aut}(\mbox{Image}(\phi))$ on the homomorphisms. It should not be hard to expand to the full set of endomorphisms.
Homomorphismswill give you the homomorphisms up to inner automorphisms, similarly to GAP'sAllHomomorphismClasses. Second, much to my personal distaste, the default setting of the parameterSurjectiveistrue. UsingG := SmallGroup( 6, 1 ); Homomorphisms( FPGroup( G ), G : Surjective := false );gives 3 homomorphisms instead of just the identity. – sTertooy Apr 24 '21 at 08:20