Your intuition that these two monads are closely related is exactly right. The difference is that Writer is much more limited, in that it doesn't allow you to read the accumulated state (until you cash out at the end). The only thing you can do with the state in a Writer is tack more stuff onto the end.
More concisely, State[S, A] is a kind of wrapper for S => (S, A), while Writer[W, A] is a wrapper for (W, A).
Consider the following usage of Writer:
import scalaz._, Scalaz._
def addW(x: Int, y: Int): Writer[List[String], Int] =
Writer(List(s"$x + $y"), x + y)
val w = for {
a <- addW(1, 2)
b <- addW(3, 4)
c <- addW(a, b)
} yield c
Now we can run the computation:
scala> val (log, res) = w.run
log: List[String] = List(1 + 2, 3 + 4, 3 + 7)
res: Int = 10
We could do exactly the same thing with State:
def addS(x: Int, y: Int) =
State((log: List[String]) => (log |+| List(s"$x + $y"), x + y))
val s = for {
a <- addS(1, 2)
b <- addS(3, 4)
c <- addS(a, b)
} yield c
And then:
scala> val (log, res) = s.run(Nil)
log: List[String] = List(1 + 2, 3 + 4, 3 + 7)
res: Int = 10
But this is a little more verbose, and we could also do lots of other things with State that we couldn't do with Writer.
So the moral of the story is that you should use Writer whenever you can—your solution will be cleaner, more concise, and you'll get the satisfaction of having used the appropriate abstraction.
Very often Writer won't give you all the power you need, though, and in those cases State will be waiting for you.