$\def\ZZ{\mathbb{Z}}$I am really baffled by people coming on this site and writing very detailed conjectures about variants of the Pepin or Lucas-Lehmer primality tests, then claiming no theoretical background. (Prior examples 1 2 3.) How did you find this conjecture if you have no idea how these tests work?
As we will see below, your test will correctly identify all cases where $8 \cdot 3^n-1$ is prime. It is quite likely it doesn't return any false positives, but I see no theoretical reason it couldn't. This involves some somewhat deep number theory. However, if I may be a bit curmedgeonly, there are also a number of elementary manipulations you could have performed at the start to make this much more readable, and I am baffled as to why you didn't.
Let's unwind your formula.
$$S_{n-1} = P_{4 \cdot 3^n}(4) = (2+\sqrt{3})^{4 \cdot 3^n} + (2-\sqrt{3})^{4 \cdot 3^n}$$
$$ = (2+\sqrt{3})^{4 \cdot 3^n} + (2+\sqrt{3})^{-4 \cdot 3^n} =(2+\sqrt{3})^{(N+1)/2} + (2+\sqrt{3})^{-(N+1)/2}.$$
You are testing whether or not $S_{n-1} \equiv 4 \bmod N$ or, on other words,
$$(2+\sqrt{3})^{(N+1)/2} + (2+\sqrt{3})^{-(N+1)/2} \equiv 4 \bmod N. \quad (\ast)$$
These are the elementary manipulations I spoke of. Surely, anyone who made this conjecture was aware of something like $(\ast)$ -- why not state it?
If $N$ is prime: (This section is rewritten to use some observations about roots of unity. It may therefore look a bit less motivated.) The prime $N$ is $-1 \bmod 24$, so $N^2 \equiv 1 \bmod 24$ and the finite field $\mathbb{F}_{N^2}$ contains a primitive $24$-th root of unity, call it $\eta$. We have $(\eta+\eta^{-1})^2 = 2 + \sqrt{3}$, for one of the two choices of $\sqrt{3}$ in $\mathbb{F}_N$. (Since $N \equiv -1 \bmod 12$, we have $\left( \frac{3}{N} \right) =1$.) Now, $\eta \not \in \mathbb{F}_N$. However, we compute $(\eta+\eta^{-1})^N = \eta^N + \eta^{-N} = \eta^{-1} + \eta$, since $N \equiv -1 \bmod 24$. So $\eta+\eta^{-1} \in \mathbb{F}_N$ and we deduce that $2+\sqrt{3}$ is a square in $\mathbb{F}_N$.
So $(2+\sqrt{3})^{(N-1)/2} \equiv 1 \bmod N$ and $(2 + \sqrt{3})^{(N+1)/2} \equiv (2 + \sqrt{3}) \bmod N$. Similarly, $(2 + \sqrt{3})^{-(N+1)/2} \equiv (2+\sqrt{3})^{-1} \equiv 2 - \sqrt{3} \bmod N$ and $(\ast)$ holds.
If $N$ is not prime. Earlier, I said that I saw no way to control whether or not $(\ast)$ held when $N$ was composite. I said that there seemed to be no reason it should hold and that, furthermore, it was surely very rare, because $N$ is exponentially large, so it is unlikely for a random equality to hold modulo $N$.
Since then I had a few more ideas about the problem, which don't make it seem any easier, but clarify to me why it is so hard. To make life easier, let's assume that $N = p_1 p_2 \cdots p_j$ is square free. Of course, $(\ast)$ holds modulo $N$ if and only if it holds modulo every $p_i$.
Let $\eta$ be a primitive $24$-th root of unity in an appropriate extension of $\mathbb{F}_{p_j}$. The following equations all take place in this extension of $\mathbb{F}_{p_j}$. It turns out that $(\ast)$ factors quite a bit:
$$(2+\sqrt{3})^{(N+1)/2} + (2+\sqrt{3})^{-(N+1)/2}=4$$
$$ (2+\sqrt{3})^{(N+1)/2} = 2 \pm \sqrt{3}$$
$$ (\eta+\eta^{-1})^{(N+1)} = (\eta+\eta^{-1})^2 \ \mathrm{or} \ (\eta^5+\eta^{-5})^2$$
$$ (\eta+\eta^{-1})^{(N+1)/2} \in \{ \eta+\eta^{-1},\ \eta^3+\eta^{-3},\ \eta^5+\eta^{-5}, \eta^7+\eta^{-7} \}. \quad (\dagger)$$
Here is what I would like to do at this point, to follow the lines of the Lucas-Lehmer test, but cannot.
(1) I'd like to know that $(\eta+\eta^{-1})^{(N+1)/2} = \eta+\eta^{-1}$, not one of the other options in $(\dagger)$. (This is what actually occurs in the $N$ prime case, as shown previously.) This would imply that $(\eta+\eta^{-1})^{(N-1)/2} =1 \in \mathbb{F}_{p_j}$.
(2) I'd like to know that the order of $\eta+\eta^{-1}$ was precisely $(N-1)/2$, not some divisor thereof.
(3) I'd like to thereby conclude that the multiplicative group of $\mathbb{F}_{p_j}$ was of order divisible by $(N-1)/2$, and thus $p_j \geq (N-1)/2$. This would mean that there was basically only room for one $p_j$, and we would be able to conclude primality.
Now, (1) isn't so bad, because you could directly compute in the ring $\ZZ/(N \ZZ)[\eta]/(\eta^8-\eta^4+1)$, rather than trying to disguise this ring with elementary polynomial formulas. So, while I don't see that your algorithm checks this point, it wouldn't be hard.
And $(2) \implies (3)$ is correct.
But you have a real problem with $(2)$. This way this works in the Lucas-Lehmer test is that you are trying to prove that $2+\sqrt{3}$ has order precisely $2^p$ in the field $\mathbb{F}_{2^p-1}[\sqrt{3}] \cong \mathbb{F}_{(2^p-1)^2}$. You already know that $(2 + \sqrt{3})^{2^p}=1$. So it is enough to check that $(2 + \sqrt{3})^{2^{p-1}}=-1$, not $1$.
In the current situation, the analogous thing would be to check that $(\eta+\eta^{-1})^{(N-1)/(2q)} \neq 1$ for every prime $q$ dividing $(N-1)/2$. But I have no idea which primes divide $q$! This seems like an huge obstacle to a proof that $(\ast)$ implies $N$ is prime.
To repeat: I think it may well be true that $(\ast)$ implies $N$ is prime, simply because there is no reason that $(\dagger)$ should hold once $N \neq p_j$, and the odds of $(\dagger)$ happening by accident are exponentially small. But I see no global principle implying this.
