Which word embedding mechanism does chatGPT use?

Question

Which word embedding mechanism does chatGPT use? Is it Word2Vec, GloVe, or something else?

Answer 1

ChatGPT is a language model based on the Transformer neural architecture, but only the decoder part.

It does not use pre-trained embeddings to represent text. Instead, the embeddings its uses are trained with the rest of the neural network. Also, the granularity ChatGPT uses to represent text is subwords, not whole words.

Its predecessor, GPT-2 is open source, so you can get an idea of ChatGPT's internals by having a look at it. GPT-3 is a "scaled up" version of GPT-2, both regarding the model and the training data. The jump from GPT-3 to GPT-3.5 (text-davinci-003) and ChatGPT is more involved than just scaling; check this article out for details.