You want to use 777 octal, not decimal:
mkdir ("/file1/file2/file3", 0777);
777 decimal turns out to be 1411 octal which will give you the bitmask 1 100 001 001 which is why you're getting those "strange" permissions. The standard set (last three segments) gives you r----x--x and the first segment modifies the world permissions to t (the sticky bit).
Also keep in mind that mkdir is subject to your umask setting and may not give you the permissions you ask for (your umask setting gets "removed" from the permissions you ask for to give you the actual permissions). See here for details, including how to avoid the problem.
You're better off using mkdir to create the directory then chmod (which isn't affected by your umask setting) to change the permissions.