A Design Partnership Case Study - ProdigalTech x Dr. Droid

About Prodigal Tech:

Prodigal Technologies, is the intelligence tool for the consumer finance ecosystem. They help turn user data into insights that can help consumer finance companies increase their revenue, reduce their cost, and minimize the risk of doing business.

Sangram Raje is the co-founder and CTO of Prodigal. Having been a quant researcher for most of his career, data has been close to him and everything that happens in the company!

The Challenge

Prodigal’s backend application has complex processes. For instance, they have a data pipeline that has long-running processes for ingestion, pre-processing and ML pipelines, all managed by different teams. (100s of GBs / day)

The challenge arises when a delay in one of the steps cascades into a poor customer experience. This smooth functioning of this whole data journey is extremely critical within their team and their data platform team owns up the end-to-end experience.

Feb-May 2023: A design partnership in making

When we started speaking, their primary requirement was to have visibility on critical pipeline status in near real-time. They had a make-shift solution internally but it was becoming tedious to manage it in-house and they were starting to explore alternate options.

Their internal solution was built over SQS-based event messaging and then added custom counter metrics in Redis, which were then monitored on a table in an internal dashboard. The challenges they had in their existing stacks were:

• Given multiple streams and in-stream processing components, the reliability of these pipelines itself was getting challenging.

• Pre-aggregating the data significantly limited the insights they were able to derive from the data.

• Inability to receive or configure alerts from their internal platform.

While they did have a roadmap of these features, and many more, when they checked out our platform, they were super excited!

Our non-tech stakeholders could directly use those dashboards to get the visibility that they wanted. One of the clear objectives that we had internally was to reduce the burden on the engineering team to give visibility to the business side because now and then we would have the business side asking engineering to give them visibility. And we were looking at Dr. Droid to solve that problem.

Niroop Belgaumkar -- Data Platform team

Implementation

Going from a POC to a company-wide implementation of Dr. Droid took them less than a quarter. Some of the key checkpoints in the implementation:

• Identifying the right checkpoints to be tracked within logs & code (~2 weeks across multiple stakeholders in ProdigalTech)

• The Prodigal team connected their Cloudwatch log streams with our connector and filtered the events relevant for them (~1 day)

• Setup for their early dashboards and metrics (~1 week)

• Adoption and awareness (~4 weeks)

Results:

Since the launch, we have had over 50% of Prodigal’s employees visit Dr. Droid dashboards. We have become their single source of truth to track the health of their pipelines (a.k.a customer happiness).

We were so happy to see the out-of-the-box functionality of Dr. Droid, which helped us set up these monitors, which were just super cool mechanisms to track all of this. Your team helped us to understand where we were making some efficiency trade-offs and helped us improve the design of our asynchronous pipelines as well. Our non-engineering folks can set up these monitors and set up alerts based on these monitors in such an intuitive way that the engineering team just doesn't need to be involved. And, most important of all, we have alerts flying out to our slack bots and searches that help us be on top of our game.

— Sangram Raje, CTO

Future Roadmap:

With Dr. Droid as Prodigal’s backbone for real-time visibility, we are working on maximising the extensibility of the platform. This means they will be able to leverage the same monitors and metrics internally as needed within their environment in case they want to set up a code runbook involving that data point.

Additionally, we are integrating with their existing observability stack (Cloudwatch, Database providers) to support the Prodigal team with automated RCA for any issue in their pipeline.

If you're interested in improving your observability, sign up today and try it out!