Arkatechture Partners with Databricks

Introduction to Arkatechture

Founded in 2012, Arkatechture is a technology company and CUSO dedicated to empowering credit unions with a better understanding of their members & organization through data. We are a team of data lovers and technical experts who use our skills to help businesses of all sizes harness, utilize, and optimize their data. In 2018, Arkatechture launched Arkalytics for Credit Unions.

Introduction to Arkalytics

Arkalytics is a fully managed, cloud-native analytics, reporting, & data science platform built on top of AWS, Tableau, and Databricks or Snowflake (our clients will now have an option to choose between Databricks and Snowflake as part of their Arkalytics implementations). At its core, Arkalytics is a Data Lakehouse with built-in business intelligence, executive & operational dashboards, and predictive models. Arkalytics not only empowers customers with a better understanding of their business through data but is also cost-effective, scalable, and highly available.

With Arkalytics, each credit union is equipped with a Data Lakehouse that ensures their teams have the most complete and up-to-date data available for data science, BI, and business analytics projects. As adoption grows, the platform now supports 26 credit unions, 100+ unique system integrations, 35+ dashboards, 3 predictive models and the flexibility to query data at differing levels of granularity and build custom content unique to each credit union.

Today, Arkatechture also partners with other fintech solutions to streamline the flow of data into, out of, and within the credit union essentially acting as the customer’s analytical and operational data hub. This puts our customers in a unique and interesting position where they are not only able to use Arkalytics for BI and analytics but also as a conduit to satisfy the data needs of their vendors. 

The “Problem”

When Arkatechture started its data journey as consulting & services partners for credit unions, we realized that most credit unions didn’t have the bandwidth or focus of an in-house data team, and those that did had a ton of unrealized value because the data platforms they had on-prem didn’t scale or were unusually expensive. Furthermore, the credit unions were spending a lot of time managing the infrastructure, wrangling data, evaluating disparate data-warehousing tools, and trying to provide meaningful insights and in doing so were constantly up against competing business priorities and regulations that the credit union engineers and analysts needed to focus on.

Statistics say that on average, 85% of projects fail when done internally, data warehouse projects cost anywhere from $500k to $2mm, and organizations normally don’t have the skills internally, and need to hire a team to support data platforms. So, they really needed to buy a data solution vs build one. This is when we introduced the credit unions to Arkalytics! Arkalytics needed to do several things for our customers:

• Provide data insight and analytics
• Remove data silos
• Be the single source of truth for all things data
• Make sure data is high quality, reliable, and up to date
• Better performance and scalability than legacy data platforms
• Be fully managed so customers could focus on business problems

Additionally, for data-savvy and data-advanced credit unions; once they are equipped with the Arkalytics data lakehouse, they want to explore expanding our data model and advancing their data analysis into developing their own data models while working alongside our data scientists. So evolving Arkalytics to solve these advanced problems and further empowering our customers to be able to fully leverage the power of their data on our platform was imperative. 

Technology

Databricks will resonate more with these data-savvy customers who are looking to extend their BI capability and enter the AI and predictive modeling space. The Databricks Lakehouse provides one platform for all data needs that are consistently governed and available for both BI and AI, built on open standards to scale efficiently with all types of workloads from simple data pipelines to massive LLMs. So, while our customers can still do BI with Databricks, they can also get the added advantage of doing predictive modeling with collaborative notebooks and managed MLflow. This gives Arkalytics the extensibility of empowering our customers, further enabling the opportunity to write their own algorithms and have a mechanism for orchestrating their models. 

One of the key aspects that Arkatechture appreciated when evaluating Databricks as a solution is its governance capabilities with Unity Catalog, and how it simplifies the ability to assign roles and create users, and cascade those roles and users to all our customers and internal employees centrally. Define once, secure everywhere. It will also help with standardization, audits and building a lineage with less administrative overhead. 

Another key feature is the Databricks integration with GIT which makes automated deployments and standing up a CI/CD pipeline very easy. This is a must-have. Model deployments are also very straightforward with Databricks. Through collaborative Databricks notebooks, managed MLflow, and Databricks Repos, model development and deployment is much faster than any other platform we have evaluated thus far.

Building on the Databricks Lakehouse Platform differentiates Arkatechture’s product by speeding up the process of implementing source system integrations because it has built-in job & scheduling infrastructure, and schema inference, which will be huge for us. It will streamline and make Arkatechture’s implementations faster from an engineering perspective. From a performance perspective, initial tests show very favorable results, likely reason to be hopeful for performance improvements. Tests using Databricks’ ZORDER function (which optimizes tables) make queries run 90% faster and, unlike clustering keys in Snowflake, it’s actually cheaper to optimize using this method. The machine learning clusters are also very fast, with similar performance to AWS Sagemaker (which we currently use)!

Partnership

Arkatechture is excited to expand Arkalytic’s offering on the Databricks Lakehouse Platform. Working with the Databricks team has allowed us to move quickly. We appreciated the whole experience of how Databricks partnered with us on the POV, providing extended support and timelines to ensure we had a solid implementation plan before the proof of value even started. The Team was very well prepared and was the exact definition of what a partnership should look like! The Databricks team wasn’t salesy, but rather focused on understanding what Arkatechture wanted from the product and if it was the right fit, for the right reasons, and the right timing. The Arkatechture team never felt pushed in a certain direction and felt like partners more than customers, which is really important to Arkatechture and its culture.

Conclusion

Arkatechture is very optimistic about what the partnership with Databricks will bring. Starting out, we see the potential benefits of using Databricks as:

• Diversified tech stack with Arkalytics (ability to offer Arkalytics on both Snowflake and Databricks without compromising on performance, scale, security or costs)
• Faster implementations and lesser administration overhead
• AI first approach and accelerating customer’s ability to build their own predictive models using Databricks notebooks
• Ability to natively build CI/CD pipelines 
• Integrating natively with change management and Arkalytics data hub
• Attaining a more data-savvy AI edge inheriting the Databricks roadmap, extending our AI capabilities
• Stream processing and real-time analytics