Here is one that I think works, and is arguably not all that unnatural.
We will start with a language along the lines of Hofstadter's BlooP. This is a simple imperative language in which the only looping construct is for loops, which require the number of iterations to be specified when the loop starts executing. There is no way to jump out of a loop conditionally. Let us say that this language has a print type statement, so that it can return results before it finishes executing.
Such a language can be simpler than the original BlooP. All you need is basic arithmetic operations (just incrementing and decrementing registers will do), some basic if type of conditional statement, and for loops and print statements as described above. If such a language is set up correctly then it is not Turing complete and can only compute primitive recursive functions. I will leave the details to the reader.
Now, you want to be able to run indefinitely, printing out results as the program runs. So let us augment this language with another looping construct, infinite_loop. This is like while (True). It enters an infinite loop and never returns, no matter what - the language should provide no way to break out of such a loop.
This is in a sense just an extension of the for loops the language already has - it's still a loop where the number of iterations has to be known in advance, it's just that now we allow an infinite number of iterations as well.
Now you can write a simple program along the lines of
n=0
infinite_loop:
print(nth_prime(n))
n += 1
where nth_prime is a (primitive recursive) subroutine that returns the nth prime number. This program satisfies your requirement of printing primes indefinitely.
However, the language provides no way of breaking out of an infinite loop conditionally. This means that all it can really do is (optionally) enter an infinite loop and then compute a primitive recursive function repeatedly, without ever returning. (You can nest infinite loops, but that doesn't change anything, since you'll never get to the second iteration of the outer one. You can store state in between invocations of your primitive recursive functions, but that doesn't change anything either, since calling a primitive recursive function n times with stored state still results in a primitive recursive function.)
So this language isn't Turing complete, but it can output prime numbers indefinitely, and it arguably isn't completely unnatural as a model of computation.