Information Retrieval (IR) has undergone a profound transformation in the field of Natural Language Processing (NLP), shifting from traditional keyword-based approaches to neural architectures and, more recently, to advanced generative Large Language Models (LLMs). Transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT) have substantially improved semantic understanding and retrieval accuracy by enabling contextualized embeddings, deeper interaction modeling, and more effective ranking mechanisms. The rise of Retrieval-Augmented Generation (RAG) represents a significant development by integrating retrieval with generation to produce factually-grounded, context-aware, and explainable outputs, while reducing the likelihood of hallucinations commonly associated with LLMs. This survey provides a comprehensive review of modern IR techniques, focusing on BERT-based retrieval models, emerging generative retrieval frameworks, evaluation methodologies, and key application domains. We provide a structured taxonomy of IR methods and conduct a comparative analysis of state-of-the-art research to highlight performance trends and methodological distinctions. Ongoing challenges, including scalability, computational efficiency, interpretability, and limitations of current evaluation benchmarks, are critically discussed. Additionally, the survey explores emerging directions such as multimodal and cross-modal retrieval, hybrid dense-sparse architectures, knowledge-graph-enhanced retrieval, and the integration of LLMs as unified retriever-generator systems. These advancements illustrate the rapid evolution of IR. They also underscore the need for adaptive, reliable, and transparent retrieval solutions in increasingly complex information environments. This work aims to provide researchers and practitioners with a clear and organized overview of the evolution, current landscape, and future research opportunities in IR within the era of LLM-driven NLP.