Pilosa, a new Austin startup that has been operating largely in stealth mode for the past year, has launched into a public beta release today.
The startup, founded by Umbel founder Higinio (H.O.) Maycotte, is available on GitHub, and it's designed to help people quickly query massive databases to accelerate a variety of work that relies on pulling insights from huge piles of information.
Pilosa's bitmap index works inside of stored information instead of on a disk, essentially turning everything into 1s and 0s. With an in-memory index separate from the actual storage, Pilosa optimizes for high cardinality data (data with a lot of unique values).
“The next wave of scientific breakthroughs will come from research projects that work with datasets of a terabyte or more,” Maycotte said. “We know how to store that data, but nobody has focused on accelerating access to that data. That changes today. Our commitment to open source ensures that this fundamental problem is solved once and for all.”
Maycotte told me that Pilosa was initially developed at Umbel, which helps drive insights from sports fan data, when a few of Umbel's biggest customers began providing datasets that other systems couldn't handle. Even Elasticsearch wasn't meeting their needs.
“We were left no option but to build our own tool.”
They saw what they had built would have a lot of applications outside of Umbel and spun the company out on its own.
Maycotte said he and his team of 13 people here in Austin decided to make Pilosa open source in order to see how many other ways the technology can be applied.
Maycottee said he's also proud Pilosa resisted the temptation to move the company to Silicon Valley and that Pilosa will stay in Austin as it moves forward to find investors and expand its offerings to include cloud and enterprise level tools to monetize and expand upon the open source solution they released today.
But, right now, Maycotte sounds most interested in Pilosa's initial use cases and whatever other developers might come up with once they access it on GitHub. Initial tests show Pilosa queries consistently fast even at high volumes and without stressing processors additionally. Pilosa says a simple query can traverse more than 2 billion edges in one second on commodity cloud hardware.
“By splitting out the database index, we think we’ll be the fastest database technology on Earth," Maycotte said.
