Operations monitoring dashboard

    Tool that provides performance and overall health monitoring capability for Azure pipelines that aims at improving data pipeline operations management
    E-Learning and Digital Library - a smiling Indian woman sitting at a table and working on her Mac computer. She is surrounded by numerous books, indicating that she is in a library. The environment appears to be quiet and conducive for studying or working.

    Problem statement

    Manual health monitoring of pipeline failures, data discrepancies and load issues leads to the reduced efficiency of the operations monitoring team. This process is resource, cost, and time intensive as the team uses multiple tools to identify the issues.

    Operations monitoring dashboard features

    Bringing efficiency and saving cost by bypassing manual effort of operations monitoring via various channels

      

    Bringing efficiency and saving cost by bypassing manual effort of operations monitoring via various channels

    Visibility and on the go availability of information to leadership and senior management on identified parameters

      

    Visibility and on the go availability of information to leadership and senior management on identified parameters

    Evaluation of performance using heterogeneous data sources at required frequency

      

    Evaluation of performance using heterogeneous data sources at required frequency

    Effective visualisation and analysis of application logs & historic data sets


      

    Effective visualisation and analysis of application logs & historic data sets

    Enable Ops stakeholders with greater analytical resources to aid decision-making

      
     

    Enable Ops stakeholders with greater analytical resources to aid decision-making

    Operations monitoring dashboard potential benefits

    • Timely detection

      Consolidated performance data to enable anomaly detection

    • Accurate detection

      Comprehensive representation to detect anomaly in a hierarchical manner 

    • Aiding resolution

      Provides historic data related to pipeline failures, their RCA and turnaround time which helps in quicker resolution.

    • Synopsis view

      Provides a summary of data flow for pipeline from source through different layers (silver, gold, etc.)

    Operations monitoring dashboard: User personas

    DesignationBusiness userOperations support & engineering leadershipPlatform operations engineer
    TaskConsumes sales data daily, to generate reports and take business decisionsResponsible for data observability to manage data quality, cost, and pipeline concernsMonitors pipelines and resolves pipeline failure issues raised by analyst or business team
    Pain pointsDue to pipeline failures, he is unable to see complete data and take business decisionsInability to timely detect failures and lack of clear visibility into failuresThe process of identifying issues with data pipeline is manual and time consuming, and requires the use of multiple tools
    Ideal experiencesLess turnaround time for resolution of pipeline issues, so that he can take decisions timely thereby not having major business impactsHolistic view of pipeline health for timely and accurate detection of failuresEasy-to-use capability that will handle monitoring and reporting. It should give precise results at required frequency and reasonable costs