in RAG, for large dataset, which similarity works? Why? how to handle problem with size of matrix in cosine similarity?

Question

If we want to implement RAG for large dataset, which similarity works? Why?

Also, how to handle problem with size of matrix in cosine similarity?