Autonomous Data Engineering
Author(s): Dinesh Thangaraju
Publication #: 2501109
Date of Publication: 10.04.2024
Country: USA
Pages: 1-12
Published In: Volume 10 Issue 2 April-2024
DOI: https://doi.org/10.5281/zenodo.14763610
Abstract
The increasing complexity of modern data systems has positioned Artificial Intelligence (AI) as a transformative force in data engineering. AI-powered tools and frameworks are streamlining data pipeline orchestration, schema creation, and quality assurance, enabling enterprises to enhance the productivity of data engineers, increase operational speed and agility, and reduce costs. However, the adoption of AI in data engineering also introduces risks related to data security, bias, and compliance that require careful management.
This paper explores how AI is reshaping data engineering, focusing on autonomous data engineering, real-time anomaly detection, and self-service analytics. It highlights the benefits of integrating AI into data workflows while addressing the associated risks. A technical framework is proposed to implement AI-driven data engineering, supported by metrics to evaluate its effectiveness.
Keywords: Data Engineering, Artificial Intelligence, Autonomous Data Engineering, Data Pipelines, Data Governance, Anomaly Detection, AI-Driven Automation
Download/View Count: 153
Share this Article