I am trying to build a Q&A bot for which I have a bunch of documents like articles (specific domain).
I understand I can create a Retrieval-Augmented Generation (RAG) system for this, but I want to know how does fine-tuning work for this case, what would be the approach here?
Would it be creating a question-answer pairs (without context) manually or automated using llms example(gpt-4) and use a pre-trained model such as LLAMA-2 to fine-tune on this QA dataset? (Creating question-answer pairs would it mean I have to create thousands of question-answer pairs that would capture almost everything about the documents I have?)
Also, if I were to pre-train the model (LLAMA-2) on the documents I have and then fine-tune on the Question-Answer (no context) , would it yield better results?
Thank you for you time in advance.
