Say I have an embedded device which is only capable of doing HMAC-SHA1 transformations (that device is, in fact, a YubiKey).
Would it be secure to feed it like a block cipher in counter mode to get a random pad suitable as an stream cipher?
Of course, I'm talking about not repeating the counter value ever to encrypt data (so this construction would also need some random IV or treating the counter value as a nonce forever).
Yes, this would be secure. CTR (Counter) mode based on keyed function $F_K$ is secure as long as its output $$ W_i = F_K(i) $$ is unpredictable given previous outputs $$ F_K(1),F_K(2),\ldots,F_K(i-1). $$ This requirement is essentially the definition of a pseudo-random function (PRF). Most HMAC instantiations with widely used hash functions are believed to be secure PRFs (rigorous proofs are impossible for several reasons).
You may have heard of CTR mode as a blockcipher mode. This is also correct, as a secure block cipher must be a secure pseudo-random permutation (PRP), which can not be distinguished from PRF if it is queried/used fewer than $2^{n/2}$ times, where $n$ is the block size.
Block ciphers are used in CTR mode primarily due to their speed (HMAC is slower, in particular on short messages), but keyed hash functions are equally strong.
Certainly, you must ensure that your counter never repeats. You may want to follow the NIST strategy (page 18) of constructing the counter from nonce.
In his book “Applied Cryptography”, Bruce Schneier says about "Ciphers Based on one-Way Hash Functions":
The simplest way to encrypt with a one-way function is to hash the previous ciphertext block concatenated with the key, then XoR the result with the current plaintext block:
$C_i = P_i \oplus H(K, C_{i-1}) \\ P_i = C_i \oplus H(K, C_{i-1})$
Set the block length equal to the output of the one-way hash function. This, in effect uses the one- way function as a block cipher in CFB mode. A similar construction can use the one-way function in OFB mode:
$C_i = P_i \oplus S_i ; S_i = H(K, C_{i-1}) \\ P_i = C_i \oplus S_i ; S_i = H(K, C_{i-1})$
The security of this scheme depends on the security of the one-way function.
This is uses the one- way function as a block cipher id OFB and CFB mode. this method can be used as stream cipher since the $C_i$ generated using XOR operation.
In relation to the “Security of Ciphers Based on one-Way Hash Functions”, Bruce Schneier states:
While these constructions can be secure, they depend on the choice of the underlying one-way hash function. A good one-way hash function does not necessarily make a secure encryption algorithm. Cryptographic requirements are different. For example, linear cryptanalysis is not a viable attack against one-way hash functions, but works against encryption algorithms. A one-way hash function such as SHA could have linear characteristics which, while not affecting its security as a one-way hash function, could make it insecure in an encryption algorithm such as MDC. I know of no cryptanalytic analysis of particular one-way hash functions as block ciphers; wait for such analysis before you trust any of them.
