Shortly after turning three years old in May, Astronomer announced a shift in focus from their software as a service product Clickstream to the open-source workflow development tool Apache Airflow.
Originating from Airbnb, Apache Airflow enables users to create data pipelines using Python, a favorite code of data scientists. It is currently being actively developed. Astronomer builds on top of Airflow make it more useable.
“Our goal as a company now is make the product the best it can be and to help organizations adopt Apache Airflow for their data infrastructure,” said Pete DeJoy, product specialist at Astronomer. “We think it’s the future of data engineering and an integral part of any big company’s data infrastructure. We’re here to help them all adopt it without any headaches.”
Astronomer was originally established in 2015 as a Clickstream data routing tool for Meteor Javascript applications. Clickstream helped companies collect, centralize, and share data behind various activities users performed with their apps and sites. During this time, Astronomer built a team to execute their mission and went through Angelpad, a top seed-stage startup incubator based in San Francisco.
One year later, the startup launched Aries Data, a data pipeline workflow library in Javascript. However, this effort was later shut down in favor of what came next.
Apache Airflow came into the picture when the Astronomer team used the tool to do custom development projects for some of their customers. The team experienced firsthand many issues with Airflow.
“We were trying to use it to develop all these data pipelines, and we’d be pulling our hair out over all the things that wouldn’t work and all the bugs that would come up,” DeJoy said. “At the same time, we had people coming up to us asking about Airflow and we became subject-matter experts. We realized it was a pretty big market opportunity to do further development on it and fix all these issues.”
However, it is difficult for an early stage company to sell two different products very well and improve upon them at the same time. Astronomer executives decided that they would rather do one thing greatly instead.
“The Airflow product had some more traction and it was more interesting to the market and investors,” DeJoy said. “It was a really exciting space to enter, so that’s where we kind of decided to run full-speed in that direction so we could focus on doing one thing in a really magnificent way.”
While many organizations, including large companies and other Cincinnati startups, are trying to adopt Airflow, they have a lot of trouble doing so because it is difficult to set up. DeJoy said Airflow is immature as a product, buggy, and still needs a lot of work to be done on it.
This is where Astronomer comes in, taking care of the infrastructural side of things with Airflow to avoid development operations headaches and provide different add-ons on top of the open-source tool.
Airflow services that Astronomer is working to provide include Airflow Podcast episodes; building and curating more plugins; a guide that facilitates building pipelines in Airflow; the Spacecamp Airflow training program; Astronomer Enterprise Edition, which allows users to spin-up and scale Apache Airflow clusters in their cloud; and an Airflow data engineering team that can be borrowed for short periods of time to power data engineering.
“I’m very proud of how we had built enough of a foundation with the team and everyone was willing to sacrifice and pull together to turn the ship around,” said Ry Walker, co-founder and CEO of Astronomer. “There’s a lot of volatility in startup life, and it’s great to have a strong team to weather the storms.”
While new details about Clickstream are still in development, Astronomer co-founder and COO Tim Brunk said in a Midwest Startups blog post that Clickstream will find a better home.
