Once you remove the two numbers $3$ and $12$, the chance of finding any 'nice math formula(s)' to solve the problem most likely plummet.
You said you were attempting to tweak some $\text{R programming language}$ code. But consider this brute force method (I am learning Python):
#--------*---------*---------*---------*---------*---------*---------*---------*
# Desc: Sum 4 addendums up to 68
#--------*---------*---------*---------*---------*---------*---------*---------*
import sys, itertools
while True:
# # create U = [1, 2, ..., 40]
U = list(range(1, 41))
# # remove 12 from U
U.pop(11)
# # remove 3 from U
U.pop(2)
countEmUp = 0
countRejects = 0
for i in itertools.product(U, U, U, U):
# # to screen for uniqueness,
# # consider standard form
if i[0] < i[1] and i[1] < i[2] and i[2] < i[3]:
pass
else:
countRejects = countRejects + 1
continue
if i[0] + i[1] + i[2] + i[3] == 68:
countEmUp = countEmUp + 1
# # print first 10 4-tuples
# # that add up to 68
if countEmUp <= 10:
print(i)
else:
countRejects = countRejects + 1
print('Combinations Adding to Sixty-Eight = ', countEmUp)
print('Rejected tuples = ', countRejects)
print(' ', '-------')
print('38 raised to the fourth power = ', 38*38*38*38)
sys.exit()
OUTPUT:
(1, 2, 25, 40)
(1, 2, 26, 39)
(1, 2, 27, 38)
(1, 2, 28, 37)
(1, 2, 29, 36)
(1, 2, 30, 35)
(1, 2, 31, 34)
(1, 2, 32, 33)
(1, 4, 23, 40)
(1, 4, 24, 39)
Combinations Adding to Sixty-Eight = 996
Rejected tuples = 2084140
-------
38 raised to the fourth power = 2085136