# The Problem of Plenty in Observability

It was quite a common issue about 5 years back to hear from teams facing the challenge that they don't have enough telemetry data to figure out how to debug a specific issue, leading to delay in investigation.

Today, the tables have turned and it's often the other way around. Most teams have over-invested in getting metrics, logs, traces and other telemetry data. But debugging and investigation still remains relatively hard.

### Reasons for accelerated adoption of observability:

Over the duration of the last decade, the adoption of monitoring & observability data collection practices have significantly improved. I'd say this has been primarily because of three reasons:

1. **Business requirements:** Increasing pace of development as most industries have accelerated towards a real-time technology world where downtimes are getting close to unacceptable. Even industries that were traditionally related to low sensitivity today have higher dependency on cloud -- for both customer facing and internal usage.
    
2. **Commercial Vendors:** Companies like Datadog & New Relic accelerated the journey for companies to get their telemetry setup. With auto-instrumentation, out of the box integrations & dashboards, these companies enabled the path to getting visibility. Additionally, a new term was coined "Observability" which evangelised & emphasised on the need to be prepared for having data to solve for issues that are unknown and un-anticipated.
    
3. **Easier Accessibility:** With adoption of observability, came the challenge of bloating costs. Logs & metrics led to costs that became a significant chunk of cloud spends, even reaching upto 25%-30% of cloud spends in some enterprises. This drove the community towards building an open source ecosystem of tooling, which is free of vendor lock-in and helps teams to manage costs. Some of the most critical projects in this regard have been OpenTelemetry, Grafana (by Grafana Labs), Prometheus and more.
    

### Is having data enough?

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1721462840559/d082c0fb-54ab-43b8-bc83-778a75340876.png align="center")

Often, doing observability right is not just about generating enough data. There are a couple of other things where you need to invest your time and energy to enable your team with a powerful tooling and debugging, at a fair cost:

**Discoverability:** Spending time to design dashboards with the right set of metrics and relevant data is as important as having the data.

**Developer Experience:** If you want your developers to be productive and fix on-call issues fast, having a good developer experience is essential. Some anti-patterns that I have noticed over the past couple of years:

* **Making data hard to access:** If you need to ssh into a jump server, then a machine, then run a bunch of kubectl commands to fetch container logs, developers might skip checking it until they check everything else.
    
* **Making data accessible but not useable:** Disabling log search by string and only allowing timeseries or a specific kv pair might be cost efficient but in case your developers need to often look at it.
    
* **Data silos:** Having telemetry data access restricted can significantly slow down the investigation process as multiple people will need to get involved to even move the needle on a task.
    
* **Docs silos:** Having docs of your team accessible to other team member could mean that they might be able to do a first level of debugging without even requiring the help of engineering team member.
    

**Starting Points & Playbooks:** Well, when the engineer is on-call and trying to fix an issue, having infinite data is more of a con than a pro. How does one actually start at the right point? So that the incident can be resolved at the earliest? Senior engineers still need to make sure they create a good way to guide on-call developers. Automated playbooks for common issues and actions taken during investigation can make it easier for on-call engineers to investigate & fix issues without escalations.

### Doctor Droid PlayBooks:

Automate investigation & remediation during production incidents by leveraging Doctor Droid. This helps significantly reduce the time it takes to investigate and fix issues.

%[https://www.youtube.com/watch?v=Uu3k-qQ3Pvw&t=1s]