1

What known state of art techniques might ChatGPT-4o, Claude 3 or other similar systems be using to understand both text and image data? I noticed that ChatGPT-4o can recognize text in an image well. Might it be using an external OCR tool or has it learned to recognize characters neurally? What training sets and methods make sense for training such text+image systems?

For some image-related tasks, such as asking to color a given image ChatGPT-4o seems to be using a less interesting approach - it looks that it generates a program and runs it on the image, the output of an image is not generated by a NN.

user163246
  • 11
  • 2

1 Answers1

1

What known state of art techniques might ChatGPT-4o, Claude 3 or other similar systems be using to understand both text and image data?

See https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation for a nice survey. Paper: https://arxiv.org/pdf/2306.13549.pdf. Also, a good VLM leaderboard: https://huggingface.co/spaces/opencompass/open_vlm_leaderboard (you can look at the best opensource VLMs).

Franck Dernoncourt
  • 5,862
  • 12
  • 44
  • 80