Encrypting small packets with low data rate

Question

I'm trying to build my own layer of encryption on top of our bluetooth communication. We can't use bonding because of some specific features we need to support. I'm a beginner when it comes to encryption, and so I wanted to run this by someone with more experience. Please look over my planned approach and point out any flaws or improvements:

1. Communication is between an iPhone and a bluetooth peripheral. At the start of any connection, the iPhone will generate a random public/private key pair. The public key will be sent to the bluetooth device.
2. The bluetooth device will generate a random symmetric key, encrypt it using the iPhone's public key and send it back to the iPhone.
3. All future communication will now be performed using the symmetric key for encryption/decryption.
4. A security passkey that is generated once for the bluetooth device and used to identify an authorized user is encrypted and sent from the iOS device to the bluetooth device. If the passkey is wrong, the connection is killed. If it is correct, the iOS device now has full access to the bluetooth device via encrypted packet communication. Each new connection will have a new private/public key pair and a new symmetric key.

Because packets are maximum 20 bytes and data rate is around 1200 bytes/second, I plan on encrypting packets using AES256 CTR encryption, as this has the advantage of the encrypted data being the same length as the original data.

Are there some flaws to my approach that I'm not aware of? Are there flaws with my plan to use CTR so the encrypted data is the same length as the original? Is there a better approach to accomplish what I want?

Edit:

I have a few questions about my specific use case.

1. I only need to keep the security passkey that is transmitted at the beginning of communications secret, as without that, an attacker wouldn't be able to do any sort of replay attack since new security keys are generated each new connection session. Is this flawed thinking?

2. Using Diffie-Hellman to generate a symmetric key, then using a stream cipher (CTR) to encrypt communication, would this be secure? Each packet would be encrypted with a new counter value (initialization vector) that would not be reused for the session, which would also be sent with the encrypted packet for decryption on the other end. My understanding is that this method would be perfectly secure as long as I never used the same initialization vector to encrypt two different packets. Is this correct thinking? My main purpose in wanting to use a stream cipher is so my encrypted data length is the same as the unencrypted data length.

3. Due to point 1 above, is it possible for me to safely get away with using the same initialization vector for all packets encrypted for the entire session, as the next session will have completely different symmetric keys, or will doing this open my protocol up to being decoded by an attacker, enabling them to aquire the passkey mentioned in point 1?

Edit2:

Based on comments by Thomas M. DuBuisson, I see that there are several issues with my approach that I hadn't thought of previously. One major issue being I need to make sure I have a solid random number generator on the bluetooth device. In light of the points he made, and after doing more research on how to securely exchange keys with a limited resource device, I thought of another approach that might be easier with the available resources, and I hope a little more secure. Sorry for my lack of experience in this area, but I'm trying to learn what I can and hopefully not make any stupid mistakes.

Answer 1

in general, a widely used algorithm to exchange/set up session keys is Diffie Hellman. While exchanging public keys, during the process a shared key is generated, that could also be used to encrypt the communication, without additonal AES.

I do not know how deep your knowledge of asymmetric encryption is- generally, there are two(three) problems that are used. None of them are trivial- factorisation, discrete logarithms and elliptic curves, the latter often combined with the first two.

However, I'm missing out some details that would help- would you rather have a fast encryption (because in that case I would recommend an elliptic-curve based Diffie Hellman, as ECDH) or an easy implementation (standard DH). Is it important that the algorithm is CCA secure(DH is not). How high shall the security be- how much computation power do you have? Can the key of the IPhone change, or should it always be the same? It would also be helpful to know what kind of bluetooth device it is.

Another approach would be using one-time keys. The communication could be initiated by the bluetooth device, which has a storage of several keys that may only be used once- in this case to encrypt the session key.

Last but not least, CTR mode is considered less secure than other cipher modes of AES. Personally, I would recommend GCM for you- because I like it, and because it also provides authenticated encryption, and can be pipelined easily. It has a 128 bit block size. If you expect many errors, OFB could be useful for you... but that goes too far. If you clarify your question a bit more as I‘ve indicated, I can help you out further.

Answer 2

I strongly suggest you use a widely accepted key-exchange technique, such as STS. I linked to my own STS specification and implementation in the comments if you are curious it might be educational.

Much of this is vague. "Start of any connection" is not specific enough for me, for example. A communication diagram would help here.

That said, here are some thoughts

1. Communication is between an iPhone and a bluetooth peripheral. At the start of any connection, the iPhone will generate a random public/private key pair. The public key will be sent to the bluetooth device.

And the device can safely trust this is the correct public key?

The bluetooth device will generate a random symmetric key, encrypt it using the iPhone's public key and send it back to the iPhone.

My current understanding is the following could happen:

Device        |    Adversary       |    Computer
E_p1(randKey) ----> E_p2(randKey2) --->

Where p1 is the adversary's public key and p2 is the computer's (or iphone's) public key.

So now the device is actually talking to the adversary in the middle, right?

All future communication will now be performed using the symmetric key for encryption/decryption.

The device uses randKey to talk to the adversary and the computer users randKey2.

A security passkey that is generated once for the bluetooth device and used to identify an authorized user is encrypted and sent from the iOS device to the bluetooth device. If the passkey is wrong, the connection is killed.

It would be wise to use a challenge-response and not a plaintext password. As I see it, the adversary now has the password (and can forward it on to the device so there is no loss of functionality visible to the user).

If it is correct, the iOS device now has full access to the bluetooth device via encrypted packet communication. Each new connection will have a new private/public key pair and a new symmetric key.

Now that you've outlined the key exchange you should also think thought this "encrypted packet communication" since that can be non-trivial as well. Is replay OK? Are out-of-order messages OK? How are you ensuring IVs are not reused? Does the key ever "die"?

A more paranoid question: Do you trust the RNG on the bluetooth device? Why? If it is responsible for all the entropy in the key then it better be great.