• 1000

Problem statement

Manual health monitoring of pipeline failures, data discrepancies and load issues leads to the reduced efficiency of the operations monitoring team. This process is resource, cost, and time intensive as the team uses multiple tools to identify the issues.

Operations monitoring dashboard features

Effective visualisation and analysis of application logs & historic data sets
Enable Ops stakeholders with greater analytical resources to aid decision-making
Evaluation of performance using heterogeneous data sources at required frequency
Bringing efficiency and saving cost by bypassing manual effort of operations monitoring via various channels
Visibility and on the go availability of information to leadership and senior management on identified parameters

Operations monitoring dashboard potential benefits

  • Timely detection - Consolidated performance data to enable anomaly detection
  • Accurate detection - Comprehensive representation to detect anomaly in a hierarchical manner 
  • Aiding resolution - Provides historic data related to pipeline failures, their RCA and turnaround time which helps in quicker resolution.
  • Synopsis view - Provides a summary of data flow for pipeline from source through different layers (silver, gold, etc.)

Operations monitoring dashboard: User personas

Designation Business user Operations support & engineering leadership Platform operations engineer
Task Consumes sales data daily, to generate reports and take business decisions Responsible for data observability to manage data quality, cost, and pipeline concerns Monitors pipelines and resolves pipeline failure issues raised by analyst or business team
Pain points Due to pipeline failures, he is unable to see complete data and take business decisions Inability to timely detect failures and lack of clear visibility into failures The process of identifying issues with data pipeline is manual and time consuming, and requires the use of multiple tools
Ideal experiences Less turnaround time for resolution of pipeline issues, so that he can take decisions timely thereby not having major business impacts Holistic view of pipeline health for timely and accurate detection of failures Easy-to-use capability that will handle monitoring and reporting. It should give precise results at required frequency and reasonable costs

Key Contacts