Fast way to convert an array of bytes representing a 24-bit image to 32-bit in Swift?

Question

I have a device that produces 24-bit BMP images and makes them available in memory as a UnsafePointer<CChar>. I need to save them as JPEG for which I'm relying on Core Image, however it doesn't support that pixel format, so I'm stupidly adding an extra byte to each pixel to make them 32-bit as follows:

// Here imageBytesPoint is the UnsafePointer<CChar> and imageSize is the width * height
let bytesPerPixel = 3
var imageBytesWithAlpha = [CChar]()
for i in stride(from: 0, to: imageSize, by: bytesPerPixel) {
  imageBytesWithAlpha.append(imageBytesPointer[i])
  imageBytesWithAlpha.append(imageBytesPointer[i + 1])
  imageBytesWithAlpha.append(imageBytesPointer[i + 2])
  imageBytesWithAlpha.append(0)
}

This is fast enough on some devices, but stupidly slow (3 seconds) on others. Is there away to optimize the loop to make it fast? As in half a second fast? I tried to port this, but the embarrassing result is even slower than my naive solution above.

(If you wonder why I'm trying to make the conversion myself, I attempted to use Core Video, but it leaks and I still cannot figure out why)