Problem statement
Manual health monitoring of pipeline failures, data discrepancies and load issues leads to the reduced efficiency of the operations monitoring team. This process is resource, cost, and time intensive as the team uses multiple tools to identify the issues.
Operations monitoring dashboard features
| Effective visualisation and analysis of application logs & historic data sets |
| Enable Ops stakeholders with greater analytical resources to aid decision-making |
| Evaluation of performance using heterogeneous data sources at required frequency |
| Bringing efficiency and saving cost by bypassing manual effort of operations monitoring via various channels |
| Visibility and on the go availability of information to leadership and senior management on identified parameters |
Operations monitoring dashboard potential benefits
- Timely detection - Consolidated performance data to enable anomaly detection
- Accurate detection - Comprehensive representation to detect anomaly in a hierarchical manner
- Aiding resolution - Provides historic data related to pipeline failures, their RCA and turnaround time which helps in quicker resolution.
- Synopsis view - Provides a summary of data flow for pipeline from source through different layers (silver, gold, etc.)
Operations monitoring dashboard: User personas
| Designation | Business user | Operations support & engineering leadership | Platform operations engineer |
|---|---|---|---|
| Task | Consumes sales data daily, to generate reports and take business decisions | Responsible for data observability to manage data quality, cost, and pipeline concerns | Monitors pipelines and resolves pipeline failure issues raised by analyst or business team |
| Pain points | Due to pipeline failures, he is unable to see complete data and take business decisions | Inability to timely detect failures and lack of clear visibility into failures | The process of identifying issues with data pipeline is manual and time consuming, and requires the use of multiple tools |
| Ideal experiences | Less turnaround time for resolution of pipeline issues, so that he can take decisions timely thereby not having major business impacts | Holistic view of pipeline health for timely and accurate detection of failures | Easy-to-use capability that will handle monitoring and reporting. It should give precise results at required frequency and reasonable costs |