Skip to Content

Advanced Analytics for Midstream Companies

Ram Seetepalli
October 31, 2022

Midstream pipeline operators continue to build out new infrastructure and modify existing infrastructure to move product from the well site to a refinery, processor, or storage facility. They are also having to continue to improve the way they manage older energy assets to further extend asset life while enhancing safety and environmental attributes.

Instrumentation and control systems have improved greatly over the past decade along with broader acceptance for remote monitoring. Midstream companies may be doing well at collecting these datasets and transmitting them, but there is still work to be done in turning all this data into actionable insights.

These companies need to continue their work on reducing the time to insight by speeding the transformation of raw data into insights to drive informed decisions.

Advanced analytics and dynamic visualization can assist a midstream company become more efficient and grow margins in some of the following ways:

  • Predictive maintenance,
  • Transportation fuel cost management,
  • Smart pressure monitoring,
  • Supply forecasting,
  • Demand forecasting,
  • Interactive operations visualization, and
  • Multi-pipeline data discovery

Predictive Maintenance

Preventive and predictive maintenance are the lifeblood of the pipeline industry.  With aging infrastructure, underwater corrosion, and other threats causing leaks or ruptures, it has become ever more important to monitor and maintain pipelines effectively to ensure that safety and security are not compromised.

Pipeline predictive maintenance has been around for a long time, but it’s only recently that it’s become much more insightful and actionable with the ability to analyze very large troves of data from past inspections and failures to maximize efficiency of production equipment and reduce downtime.

One of the common causes of incidents in oil pipelines is defective pipe walls. Prediction of pipeline’s remaining lifetime can be estimated by using consistent assessments of inspection data such as corrosion rates at specific cross sections. Repair actions are then executed on specific cross-sections of the pipeline.

Historically, very little of the overall collected data was used for analysis. Machine learning and dynamic visualization can help render opaque mounds of inspection data replicating the actual pipe over the life of the asset.  Rather than focusing on isolated data points in pipeline inspection data sets, interacting threats can be modeled by correlating anomalies. This may extend to data beyond a single pipeline to those that have similar properties and location data.

Predictive maintenance with machine learning allows pipeline operators to take proactive actions, increasing operational efficiency, life of pipeline and reducing chances of faults and unplanned shutdowns.

Transportation Fuel Cost Management

Given the high number of variables that are involved in the ongoing analysis of pipeline portfolios, it can be challenging to maximize transportation fuel efficiency. Variables such as ambient and ground temperatures, pipeline pressure, and the volume of gas flow are interrelated.

Majority of oil pipeline rates are based on the oil pipeline rate index approved by the FERC, which is adjusted regularly. As the index changes, companies must be able to optimize fuel consumption per unit of transported product and properly correlate all key data points. With the appropriate analytical models that can be rapidly selected, trained and tested, they have more confidence in this optimization, because they can base their analysis on a combination of real-time SCADA data and many years of historical data from prior pipeline operations.

Smart Pressure Monitoring

Pipeline pressures regularly fluctuate. Most pressure drops are related to normal activities such as power plants drawing fuel or chemical plants ramping up production. Other pressure drops could be the result of anomalous conditions, such as ruptures or leaks. Regulators press pipeline managers to control risk posed by those fluctuations by building automated systems that can both accurately detect and respond to anomalies.

SCADA systems can monitor these pressure drops, but they are not well suited to interpreting these occurrences and categorizing them as either standard or anomalous fluctuations.

With reactive programming, real-time alerts can be generated based on stream analysis of the SCADA data. These alerts can be simple rule-based algorithms or could be very complex machine learning algorithms. Further, the machine learning models can greatly assist in identifying relevant events and triggering contextual and timely notifications.

Supply-Side Forecasting

Producers allocate specific gas volumes for transportation to markets one day prior to delivery. They base those allocations on the current production from gas wells. These “day ahead” predictions are subject to great uncertainty because of corresponding uncertainty in the gas production process. It is difficult to know in advance from which location the gas will enter, at what volumes, and at what pressure.

This can leave midstream companies scrambling to configure the pipeline to manage volumes different from what they expected. This often results in a sub-optimal configuration and reduced revenues from the asset.

While uncertain, producer volumes may be subject to repeatable patterns and if so, a proper forecasting algorithm could provide useful insight and guidance as to expected volumes from producers. Moreover, analytical models can reveal ways in which midstream companies might use pricing incentives that induce producers to smooth volumes for their own benefit.

Demand-Side Forecasting

Like supply-side forecasting, demand-side contains some inherent uncertainties, particularly with gas-fired power plants attached to interstate pipelines. The electricity market may provide signals that can be correlated to activity at these power plants and linked to future demand for gas. Machine learning models give producers longer-term sense-and-respond capabilities. These insights allow midstream companies to more effectively balance and further optimize the pipeline.

Interactive Operations Visualization

Midstream companies produce and distribute comprehensive reports on pipeline condition and activities. These reports typically contain adds, withdrawals, and storage system activity–an operational snapshot of the business from the previous day.

While the reports contain extremely useful information, they provide a static view of a dynamic operation and so do not efficiently serve the needs of some stakeholders. The facts and the insights typically delivered in these reports can be more efficiently converted to real-time or near real-time interactive visualizations leveraging gaming engines like Unreal or Unity. These dynamic visualizations can also very efficiently playback significant events and serve actionable views to the various decision makers.

Data Discovery and Exploration Across Multiple Pipelines

Midstream companies attempt to optimize revenue and operating costs across multiple pipeline systems. This requires analysis for the transportation, operations, finance and other teams in those enterprises.

Capturing data from all the pipelines and combining that into a single multi-pipeline dataset provides the foundation necessary for exploratory data discovery with the ability to build statistical and analytical models across pipelines that drives better efficiency.


This is a small set of use cases for driving efficiency and there are many more cases for better securing, maintaining and running daily operations with advanced analytics.

Forward-thinking, innovative midstream companies can take advantage of mature analytics practices, tools and platforms to process the unprecedented speed and volume of data. Emerging types of data, such as machine and sensor data, geolocation data, weather data, and log data become valuable at high volumes, especially when correlated with other historical and enterprise data sets.

The patterns within these large troves of data fuel machine learning applications designed to better understand and analyze many critical aspects of a midstream company’s operations.

About the author

Ram is a Technology Executive with over 20 years of experience in defining technology and innovation across industries. He has consulted as an advisor to several Fortune 100 organizations on leveraging innovation and technology to deliver measurable business value.


    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Slide to submit