Building a Dynamic Data Ingestion Framework to Manage Schema Evolution for an Enterprise

Author(s)	Shreesha Hegde Kukkuhalli
Country	USA
Abstract	In the era of big data, the dynamic nature of data sources has led to increased complexity in data ingestion frameworks, especially when managing schema evolution. Traditional data ingestion pipelines often fail to adapt effectively to changing data structures, leading to data loss, delays, and errors. This paper proposes a novel dynamic data ingestion framework designed to handle schema evolution seamlessly in real-time. This framework leverages schema inference, metadata-driven transformations, and automated version control to support heterogeneous data sources, reduce latency, and enhance data accuracy. Our experimental results show significant improvements in adaptability and data integrity compared to traditional ingestion architectures
Keywords	Data Ingestion, Schema Evolution, Dynamic Framework, Big Data, Metadata-Driven, Real-Time Processing, Automated Version Control, ETL.
Published In	Volume 4, Issue 2, March-April 2022
Published On	2022-03-25
Cite This	Building a Dynamic Data Ingestion Framework to Manage Schema Evolution for an Enterprise - Shreesha Hegde Kukkuhalli - IJFMR Volume 4, Issue 2, March-April 2022. DOI 10.36948/ijfmr.2022.v04i02.12014
DOI	https://doi.org/10.36948/ijfmr.2022.v04i02.12014
Short DOI	https://doi.org/g82h4w

About IJFMR Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijfmr.com

International Journal For Multidisciplinary Research