One of the recurring peculiarities of oil complex data as reported by the EIA was how, during a time of an unprecedented crude glut by OPEC and pronounced economic weakness in the US, was overall US demand of various petrochemical products as strong as the DOE reported on a weekly basis. To be sure, the alleged increase in demand was one of the major catalysts that prompted rising oil prices together with relentless jawboning by OPEC members about a "production freeze" that would never materialize, in turn spurring not one but two record short squeeze across the commodity complex.
We now know the answer.
In a note released moments ago by the EIA, whose bias to keeping prices as high as possible is no secret, admitted that "over the first six months of 2016, EIA weekly estimates underestimated total crude oil, petroleum, and biofuel exports by an average of 16%, compared with final data published in the PSM."
This underestimation of exports "led to the overestimation of total consumption" by a similar amount. The new methodology using near-real-time data from Customs significantly reduces the difference between weekly estimates and the actual data for total exports shown in the PSM during the first half of 2016.
So time to fix the mistake then, and as a result, the EIA said that starting with today's release of the Weekly Petroleum Status Report (WPSR), EIA is now publishing weekly petroleum export and consumption estimates based on near-real-time export data provided by U.S. Customs and Border Protection (Customs). EIA previously relied on weekly export estimates based on monthly official export data published by the U.S. Census Bureau roughly six weeks following the end of each reporting month. This new methodology is expected to improve weekly estimates of petroleum consumption (measured as product supplied) by improving estimates of weekly exports of crude oil, petroleum products, and biofuels, which increased from 1 million barrels per day (b/d) in 2004 to nearly 5 million b/d in 2015.
The EIA adds that the use of near-real-time export data should reduce differences between EIA's weekly data, as presented in the WPSR, and monthly data, as presented in the Petroleum Supply Monthly (PSM). The monthly data that EIA publishes 60 days after the end of each month are based on EIA's comprehensive monthly survey data and the actual Census Bureau export data for that month.
As the EIA adds, the difference between the old and new weekly methodologies differs across individual products, with the new methodology providing a particularly significant improvement in the estimate of finished motor gasoline exports for the first six months of 2016. The improvement in export values reduces the difference in finished motor gasoline consumption from within 1.3% to within 0.9% of the actual values published in the Petroleum Supply Monthly for the first six months of 2016.
Still, don't assume that all the bias will be eliminated: while the new weekly export methodology should provide improved weekly petroleum consumption estimates, there still may be differences between the weekly and monthly balances. Although the Census Bureau is able to directly validate data with export filers, EIA processes the raw Customs data without the ability to directly validate reported data with those filers and adds estimates for data not reported by Customs. In addition, there will continue to be minor differences between weekly sampled survey data for the nonexport categories and monthly data because of standard sampling and statistical issues.
The EIA concludes its mea culpa by saying that "historically, these differences have been minor, with the export data being the largest source of disconnect between the two series."
What the EIA did not say is that for HFT funds using headline scanning algos in which even the smallest "beat" of expectations is sufficient to launch a market-wide momentum igniting stop hunt in Brent and WTI, these "minor" disconnects could have been - and were - sufficient to reprice the oil complex substantially higher. Alas, we doubt the same HFTs will now correct for months of incorrect data.