(nmedia/Shutterstock)
Organizations have been engaged on turning into extra data-driven for a few years at this level, with blended outcomes. We perceive that the worth of knowledge is simple. Nonetheless, it has now develop into extra essential than ever for organizations to perform as data-driven entities. The urgency of this difficulty is underscored by the emergence of AI and ML.
The 2024 State of Apache Airflow helps us perceive the expansion, utilization, and consumer neighborhood dynamics of navigating the complexities of knowledge orchestration, information pipelines, and information supply.
The info for the report was compiled by surveying 281 professionals who had been immediately or not directly liable for managing their group’s Airflow situations. This included information engineers, software program engineers, DevOps engineers, and options architects.
The report highlights three vital traits – fast development in Airflow adoption, rising reliability of Airflow for mission-critical information supply, and Airflow turning into the usual for AI pipelines.
Apache Airflow is an open-source platform for creating, scheduling, and monitoring information and computing workflows. With its graphical UI, integration with main cloud platforms, and ease of use, Airflow is turning into a preferred software for orchestrating workflows or pipelines.
One of many key findings of the report is that AL and ML services by Airflow spiked by 24 % 12 months over 12 months. The highest purposes for Airflow included the inner analytics dashboard, GenAI, and ML. There has additionally been a 68 % improve in Airflow downloads 12 months on 12 months with over 165.7 million downloads. Airflow contributors outpace Apache initiatives Spark and Kafka with 2.8K contributors
Whereas the AI market is projected to proceed rising, there are some considerations, together with fragmented software stacks, siloed instruments, and a looming menace of stricter compliance and operational necessities.
In accordance with Apache, a rising variety of companies are addressing these considerations utilizing Astronomer – the unified information platform constructed on Apache Airflow service. Astronomer permits organizations to make sure information privateness, and unify information throughout the cloud, staff, and deployments. It additionally helps speed up improvement by seamless integrations.
(Andrey Suslov/Shutterstock)
The report reveals that Apache Airflow has developed from a distinct segment software to a mission-critical asset. Two-thirds (67 %) of corporations have greater than 6 individuals utilizing Airflow. This means a rising pattern of knowledge groups utilizing Airflow to ship enterprise outcomes.
It’s not simply the variety of individuals utilizing Airflow that’s spectacular, the frequency of interplay can also be excessive. The report exhibits that 55 % of customers work together with Airflow each day, with one other 26 % utilizing it at the very least as soon as every week. It seems that corporations are rising of their confidence in Airflow and their imaginative and prescient of what the platform can accomplish.
Round half (46 %) of respondents shared that they really feel Airflows is a “crucial” software for his or her enterprise operations. Outages and the necessity for scalability had been the highest two considerations of the respondents. The respondents additionally shared that probably the most regarding facets of knowledge downtime embrace inner techniques being negatively affected, throttling of knowledge staff productiveness, and points with revenue-generation apps or providers.
The outcomes of the survey spotlight how Apache Airflow meets the rising challenges of knowledge orchestration, scaling, and facilitation of greatest practices. Maybe the best influence of Airflow has been in AI pipeline supply. The momentum positions Airflows as one of many main information orchestration platforms for AI improvement and deployment.
Associated Objects
How Airflow 2.8 Makes Constructing and Operating Information Pipelines Simpler
Astronomer’s Apache Airflow-Powered Astro Outperforms Opponents with 30M Month-to-month Downloads
AWS Pronounces Basic Availability of Amazon Managed Workflows for Apache Airflow