This page has been archived and commenting is disabled.

Scientific Proof That High Frequency Trading Induces Adverse Changes In Market Microstructure And Dynamics, And Puts Market Fairness Under Question

Tyler Durden's picture




 

Up until recently, any debate between proponents and opponents of High Frequency Trading would typically be represented by heated debates of high conviction on either side, with discussions rapidly deteriorating into ad hominem attacks and the producer screaming 'cut to commercial' to prevent fistfights. Luckily, all this is about to change. In a research paper by Reginald Smith of the Bouchet Franklin Institute in Rochester titled "Is high-frequency trading inducing changes in market microstructure and dynamics?" the author finds that he "can clearly demonstrate that HFT is having an increasingly large impact on the microstructure of equity trading dynamics. Traded value, and by extension trading volume, fluctuations are starting to show self-similarity at increasingly shorter timescales. Values which were once only present on the orders of several hours or days are now commonplace in the timescale of  seconds or minutes. It is important that the trading algorithms of HFT traders, as well as those who seek to understand, improve, or regulate HFT realize that the overall structure of trading is influenced in a measurable manner by HFT and that Gaussian noise  models of short term trading volume fluctuations likely are increasingly inapplicable." In other words, the author finds ample evidence that during the past decade (on the NASDAQ) and especially since the 2005 revision of Reg NMS (on the NYSE), stock trading increasingly demonstrates "self similar" fractal patterns, resulting in volatility surges, recursive feedback loops, and a market structure which is increasingly becoming a product of the actual trading mechanism. In the process, as demonstrated by a Hurst Exponent gravitating increasingly further away from 0.5 (i.e., Brown Noise territory), the Markov Process nature of stock trading is put under question, and thus the whole premise of an efficient market has to be reevaluated. Simply said: HFT has been shown to affect the fairness of trading.

The paper is, needless to say, a must read for everyone who has an even passing interest in stock trading and market regulation (alas, yes, that would mean the SEC, and Congress). And while one of the key qualities of the paper is presenting the history and implications of High Frequency Trading, and its rise to market dominance primarily as a result of the revision of Reg NMS, allowing stock trading to become a free for all for every algo, and ECN/ATM imaginable, the key findings are what makes it unique. In analyzing stochastic processes and fractal phenomena, and concluding that the Hurst Exponent of transactions that involve less than 1,500 shares per trade (and especially less than 250 - a distinct subdomain relegated to HFT strategies) is no longer 0.5, the author validates the skepticism of all those who for over a year (such as Zero Hedge) have claimed that the direct and increasing involvement of HFT is an de-evolutionary process that is leading to increasing market fragmentation, self-sameness, destabilization, and volatility, offset merely by allegedly improved liquidity, which incidentally disappears on a moment's notice when the negative side-effects of HFT overwhelm the positive, such as was the case on May 6. Furthermore, the authors find that the type of fractal recursive feedback loops inspired by increasing HFT participation lead to spikes in correlation: "Correlations previously only seen across hours or days in trading time series are increasingly showing up in the timescales of seconds or minutes." And due to the implied fractal nature of trading (think standing waves, fern leaves, sandy beaches, and all other goodies unleashed upon the world courtesy of Benoit Mandelbrot), it appears investors now have to consider such quixotic issues as Lorenzian Attractors when it comes to simple trading. What is most troubling, is that micro similarities, as postulated by non-linear theory, tend to rapidly evolve into massively scaled topological disturbances, and thus a few simple resonant trades can rapidly avalanche into a major market destabilizing event.... such as that seen on May 6.

While the math of the article is a little daunting, and the author appears to derive a peculiar satisfaction from throwing the Riemann Zeta function in the general mathematical stew (incidentally, with the prevalent IQ of Zero Hedge readers being sufficiently high to allow at least a valiant effort at proving the Riemann Hypothesis, we hope some of our more industrious readers take it upon themselves to venture and pocket the generous proceeds from the Millennium Prize, for which we will be content to receive a mere pittance as a donation for proffering this forum), the observations and conclusions are water tight:

Given the complex nature of HFT trades and the frequent opacity of firm trading strategies, it is difficult to pinpoint exactly what about HFT causes a higher correlation structure. One answer could be that HFT is the only type of trading that can exhibit trades that are reactive and exhibit feedback effects on short timescales that traditional trading generates over longer timescales.

Another cause may be the nature of HFT strategies themselves. Most HFT strategies can fall into two buckets Lehoczky and Schervish (2009):

(i) Optimal order execution: trades whose purpose is to break large share size trades into smaller ones for easier execution in the market without affecting market prices and eroding profit. There are two possibilities here. One that the breaking down of large orders to smaller ones approximates a multiplicative cascade which can generate self-similar behavior over time Mandelbrot (1974). Second, the queuing of chunks of larger orders under an M/G/1 queue could also generate correlations in the trade flow. However, it is questionable whether the “service time”, or time to sell shares in a limit order, is a distribution with infinite variance as this queuing model requires.

(ii) Statistical arbitrage: trades who use the properties of stock fluctuations and volatility to gain quick profits. Anecdotally, these are most profitable in times of high market volatility. Perhaps since these algorithms work through measuring market fluctuations and reacting on them, a complex system of feedback based trades could generate self-similarity from a variety of yet unknown processes.

Since firm trade strategies are carefully guarded secrets, it is difficult to tell which of these strategies predominate and induces most of the temporal correlations.

When it comes to the interplay of optimal order execution and statistical arbitrage, it can easily be seen why large block splitting into child orders could conceive a self-similar trading pattern that reverberates across the market, in an increasingly micro-correlated and fractal marketplace. In the course of events on May 6, it is perfectly feasible that as many mutual funds commenced dumping large blocks of stock, assorted algorithms had to work overtime to split these orders into millions of small trade blocks. And with statistical arbitrage models programmed to game and front-run large order blocks by diving the intentions of repeated micro orders, it becomes all too clear how a rapid selling event can rapidly culminate into a bid-less environment where both the stat arb and order execution HFT algorithms are all on the same side of the boat. Consider the market action from the past several days as indicative of micro volume accumulation by HFTs, which is only offset by mega volume dumping - once all the HFTs are forced to unwind and go to cash, the actual principal liquidity providers who in their desperation become liquidity takers, suddenly find themselves with no recourse but to hit any bid. Which is why the NYSE explanation of Liquidity Replenishment Points is nothing but complete BS - the market meltdown had nothing to do with selective order routing to non-NYSE venues, and everything to do with a fractal implosion, in which, as Nassim Taleb would explain, the Hurst Exponent briefly went from 0.5 to infinity minus 1, and the entire market became correlated with itself.

Of course if the paper is correct, and the empirical evidence presented in it is sufficient to eliminate doubt, it means that in the coming years we will have an exponentially growing number of days in which May 6-type event will recur over and over.

The paper's conclusion:

Given the above research results, we can clearly demonstrate that HFT is having an increasingly large impact on the microstructure of equity trading dynamics. We can determine this through several main pieces of evidence. First, the Hurst exponent H of traded value in short time scales (15 minutes or less) is increasing over time from its previous Gaussian white noise values of 0.5. Second, this increase becomes most marked, especially in the NYSE stocks, following the implementation of Reg NMS by the SEC which led to the boom in HFT. Finally, H > 0.5 traded value activity is clearly linked with small share trades which are the trades dominated by HFT traffic.

In addition, this small share trade activity has grown rapidly as a proportion of all trades. The clear transition to HFT influenced trading noise is more easily seen in the NYSE stocks than with the NASDAQ stocks except NWSA. The main exceptions seem to be GENZ and GILD in the NASDAQ which are less widely traded stocks. There are values of H consistently above 0.5 but not to the magnitude of the other stocks. The electronic nature of the NASDAQ market and its earlier adoption of HFT likely has made the higher H values not as recent a development as in the NYSE, but a development nevertheless.

Given the relative burstiness of signals with H > 0.5 we can also determine that volatility in trading patterns is no longer due to just adverse events but is becoming an increasingly intrinsic part of trading activity. Like internet traffic Leland et. al. (1994), if HFT trades are self-similar with H > 0.5, more participants in the market generate more volatility, not more predictable behavior.

There are a few caveats to be recognized. First, given the limited timescale investigated, it is impossible to determine from these results alone what, if any, long-term effects are incorporating the short-term fluctuations. Second, it is an open questions whether the benefits of liquidity offset the increased volatility. Third, this increased volatility due to self-similarity is not necessarily [TD: but very well could be as described above] the cause of several high profile crashes in stock prices such as that of Proctor & Gamble (PG) on May 6, 2010 or a subsequent jump (which initiated circuit breakers) of the Washington Post (WPO) on June 16, 2010. Dramatic events due to traceable causes such as error or a rogue algorithm are not accounted for in the increased volatility though it does not rule out larger events caused by typical trading activities. Finally, this paper does not investigate any induced correlations, or lack thereof, in pricing and returns on short timescales which is another crucial issue.

Traded value, and by extension trading volume, fluctuations are starting to show self-similarity at increasingly shorter timescales. Values which were once only present on the orders of several hours or days are now commonplace in the timescale of seconds or minutes. It is important that the trading algorithms of HFT traders, as well as those who seek to understand, improve, or regulate HFT realize that the overall structure of trading is influenced in a measurable manner by HFT and that Gaussian noise models of short term trading volume fluctuations likely are increasingly inapplicable.

As for evidence, we refer readers to the paper itself, but in a nutshell, the authors find that over the years, on both the NYSE (after Reg NMS revision in 2005) and on the Nasdaq (from before, as the Nasdaq was the original spawn of HFT strategies), as the prevalent share bucket moved from greater than 1,000 shares per trade, to 250 or less, direct evidence of increasing HFT dominance, especially coupled with previous Tabb group evidence that over 70% of all trading is conducted by HFT, the Hurst Exponent of all trading increasingly moved away from 0.5, and has hit as high as 0.7 in some case: a stunning result which puts the entire stochastic nature of stock markets in question! (see charts below).

Charting the average size per trade since 2002:

And charting the Hurst Exponent as calculated by the authors in a variety of Nasdaq and NYSE stocks:

 

We are confident that this paper will serve as the guiding light to much more comparable research, due to the unique approach the author takes in analyzing stock behavior. In moving away from a traditional and simplistic Gaussian frame, Smith isolates the very nature of the problem, which like any other non-linear system, and thus prone to Black Swanness, has to be sought in the plane of fractal geometry. Luckily, the author provides the one elusive observation which many market participants (at least those whose livelihoods are not tied into the perpetuation of the destructive HFT processes) had long sensed was on the tips of their tongues, yet the only comprehensible elucidation was the trite and overworn "the market is broken." At least now we know that this is a fact.

Unfortunately, as the paper requires slightly more than first grade comprehension and math skills, it will never be read by anyone at the SEC, or those in Congress, who are pretending to be conducting Financial Regulation Reform, when the items described in this paper are precisely the things that any reform should be addressing.

And while we again urge everyone to read the full paper, below we present the section of the paper that does a terrific job in explaining the arrival of HFT, its development over the ages, and its parasitic role in market structure.

Appendix:

A brief history of the events leading up to high frequency trading

In 1792, as a new nation worked out its new constitution and laws, another less heralded revolution began when several men met under a Buttonwood tree, and later coffee houses, on Wall St. in New York City to buy and sell equity positions in fledgling companies. An exclusive members club from the beginning, the New York Stock Exchange (NYSE) rapidly grew to become one of the most powerful  exchanges in the world. Ironically, even the non-member curbside traders outside the coffee houses gradually evolved into over-the-counter (OTC) traders and later, the NASDAQ. A very detailed and colorful history of this evolution is given in Markham and Harty (2008); Harris (2003).

Broadly, the role of the exchange is to act as a market maker for company stocks where buyers with excess capital would like to  purchase shares and sellers with excess stock would like to liquidate their position. Several roles developed in the NYSE to encourage smooth operation and liquidity. There came to be several types of market makers for buyers and sellers known as brokers, dealers, and specialists. The usual transaction involves the execution of a limit order versus other offers. A limit order, as contrasted to a market order which buys or sells a stock at the prevailing market rate, instructs the purchase of a stock up to a limit ceiling or the sale of a stock down to a limit floor. Brokers act on behalf of a third-party, typically an institutional investor, to buy or sell stock according to the pricing of the  limit order. Dealers, also known as market-makers, buy and sell stock using their own capital, purchasing at the bid price and selling at the ask price, pocketing the bid-ask spread as profit. This helps ensure market liquidity. A specialist acts as either a broker or dealer but only  for a specific list of stocks that he or she is responsible for. As a broker, the specialist executes trades from a display book of outstanding orders and as a dealer a specialist can trade on his or her own account to stabilize stock prices.

The great rupture in the business-as-usual came with the Great Depression and the unfolding revelations of corrupt stock dealings, fraud,  and other such malfeasance. The Securities and Exchange Commission (SEC) was created by Congress in 1934 by the Securities Exchange Act. Since then, it has acted as the regulator of the stock exchanges and the companies that list on them. Over time, the SEC and Wall Street have evolved together, influencing each other in the process.

By the 1960s, the volume of traded shares was overwhelming the traditional paper systems that brokers, dealers, and specialists on the floor used. A“paperwork crisis” developed that seriously hampered operations of the NYSE and led to the first electronic order routing system, DOT by 1976. In addition, inefficiencies in the handling of OTC orders, also known as “pink sheets”, led to a 1963 SEC recommendation of changes to the industry which led the National Association of Securities Dealers (NASD) to form the NASDAQ in 1968. Orders were displayed electronically while the deals were made through the telephone through“market makers” instead of dealers or specialists. In 1975, under the prompting of Congress, the SEC passed the Regulation of the National Market System, more commonly known as Reg NMS, which was used to mandate the interconnectedness of various markets for stocks to prevent a tiered marketplace where small, medium, and large investors would have a specific market and smaller investors would be disadvantaged. One of the outcomes of Reg NMS was the accelerated use of technology to connect markets and display quotes. This would enable stocks to be traded on different, albeit connected, exchanges from the NYSE such as the soon to emerge electronic communication networks (ECNs), known to the SEC as alternative trading systems (ATS).

In the 1980s, the NYSE upgraded their order system again to SuperDot. The increasing speed and availability computers helped enable trading of entire portfolios of stocks simultaneously in what became known as program trading. One of the first instances of algorithmic trading, program trading was not high-frequency per se but used to trade large orders of multiple stocks at once. Program trading was  profitable but is now often cited as one of the largest factors behind the 1987 Black Monday crash. Even the human systems broke  down, however, as many NASDAQ market makers did not answer the phones during the crash.

The true acceleration of progress and the advent of what is now known as high frequency trading occurred during the 1990s. The telecommunications technology boom as well as the dotcom frenzy led to many extensive changes. A new group of exchanges became prominent. These were the ECN/ATS exchanges. Using new computer technology, they provided an alternate market platform where buyers and sellers could have their orders matched automatically to the best price without middlemen such as dealers or brokers. They  also allowed complete anonymity for buyers and sellers. One issue though was even though they were connected to the exchanges via Reg NMS requirements, there was little mandated transparency. In other words, deals settled on the ECN/ATS were not revealed to the  exchange. On the flip side, the exchange brokers were not obligated to transact with an order displayed from an ECN, even if it was  better for their customer.

This began to change, partially because of revelations of multiple violations of fiduciary duty by specialists in the NYSE. One example,  similar to the soon to be invented ‘flash trading’, was where they would “interposition” themselves between their clients and the best  offer in order to either buy low from the client and sell higher to the NBBO (National Best Bid and Offer; the best price) price or vice versa.

In 1997, the SEC passed the Limit Order Display rule to improve transparency that required market makers to include offers at better  prices than those the market maker is offering to investors. This allows investors to know the NBBO and circumvent corruption.

However, this rule also had the effect of requiring the exchanges to display electronic orders from the ECN/ATS systems that were better priced. The SEC followed up in 1998 with Regulation ATS. Reg ATS allowed ECN/ATS systems to register as either brokers or exchanges. It also protected investors by mandating reporting requirements and transparency of aggregate trading activity for ECN/ATS systems once they reach a certain size.

These changes opened up huge new opportunities for ECN/ATS systems by allowing them to display and execute quotes directly with the  big exchanges. Though they were required to report these transactions to the exchange, they gained much more. In particular, with their  advanced technology and low-latency communication systems, they became a portal through which next generation algorithmic trading  and high frequency trading algorithms could have access to wider markets. Changes still continued to accelerate.

In 2000, were two other groundbreaking changes. First was the decimalization of the price quotes on US stocks. This reduced the bid-ask spreads and made it much easier for computer algorithms to trade stocks and conduct arbitrage. The NYSE also repealed Rule 390 which had prohibited the trading of stocks listed prior to April 26, 1979 outside of the exchange. High frequency trading began to grow rapidly but did not truly take off until 2005.

In June 2005, the SEC revised Reg NMS with several key mandates. Some were relatively minor such as the Sub-Penny rule which  prevented stock quotations at a resolution less than one cent. However, the biggest change was Rule 611, also known as the Order Protection Rule. Whereas with the Limit Order Display rule, exchanges were merely required to display better quotes, Reg NMS Rule 611 mandated, with some exceptions, that trades are always automatically executed at the best quote possible. Price is the only issue and not counterparty reliability, transaction speed, etc. The opening for high frequency trading here is clear. The automatic trade execution created the perfect environment for high speed transactions that would be automatically executed and not sit in a queue waiting for approval by a market maker or some vague exchange rule. The limit to trading speed and profit was now mostly the latency on electronic trading systems.

The boom in ECN/ATS business created huge competition for exchanges causing traditional exchanges (NYSE & Euronext) to merge and  some exchanges to merge with large ECNs (NYSE & Archipelago). In addition, the competition created increasingly risky business strategies to lure customers. CBSX and DirectEdge pioneered ‘flash trading’ on the Chicago Board of Exchange and the NYSE/NASDAQ  respectively where large limit orders would be flashed for 50 milliseconds on the network to paying customers who could then rapidly trade in order to profit from them before public advertisement. Many of these were discontinued in late 2009 after public outcry but HFT was already the dominant vehicle for US equity trading as shown in figure 1. HFT thrives on rapid fire trading of small sized orders and the overall shares/trade has dropped rapidly over the last few years is shown in figure 2. In addition, the HFT strategy of taking advantage of pricing signals from large orders has forced many orders off exchanges into proprietary trading networks called ‘dark pools’ which get their name from the fact they are private networks which only report the prices of transactions after the transaction has occurred and typically anonymously match large orders without price advertisements. These dark pools allow a safer environment for large trades which (usually) keep out opportunistic high frequency traders. The basic structure of today’s market and a timeline of developments are given in figure 3 and figure 4. For more detailed information, see Stoll (2006); McAndrews and Stefandis (2000); Francis et. al. (2009); Mittal (2008); Degryse et. al. (2008); Palmer (2009)

 

Full paper link here.

 

- advertisements -

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Tue, 07/13/2010 - 00:54 | 465781 thesapein
thesapein's picture

Wait, what? Only now are financial analyst applying fractal math? Man, this field is wide open for some seriously cool science if experts in finance and trade really see this as novel stuff. Then again, they're still treating Fibonacci as new thing, so, yeah, time for some upgrades. 

Tue, 07/13/2010 - 01:37 | 465817 currency confla...
currency conflagration's picture

Maybe Apple can help them out with a fractal app!

Tue, 07/13/2010 - 01:53 | 465826 thesapein
thesapein's picture

(Just using you (cc) as a stepping stone and being honest about it...)

You know, from a nube's perspective, it kind of looks like a duh moment because, of course, if the time frame is shortened, why wouldn't we expect to see a self similar pattern? HFT allows for a more dense curve, or more points within a frame, so unless HFT uses different rules it should show, after zooming in, a similar curve, just on a different scale, no?

Also, Tyler, give yourself credit for already shining some light on the dominance in the market of HFT before this paper was written. 

Tue, 07/13/2010 - 02:07 | 465836 ThinkAVP
ThinkAVP's picture

Tyler, CRUSHHHHING IT!!!! one post at a time..

 

"ON a long enough timeline the survival rate for everyone comes to Zero Hedge"

Tue, 07/13/2010 - 02:14 | 465841 thesapein
thesapein's picture

I feel like throwing a ZH party.

Tue, 07/13/2010 - 02:18 | 465845 ThinkAVP
ThinkAVP's picture

LET'S CRUSHH IT!!  I'll Bring the ZH chicks..

Tue, 07/13/2010 - 02:24 | 465851 ThinkAVP
ThinkAVP's picture

Tyler, I need a ZH Application for my IPhone,  ill gladly contribute to the CREATION!!

Tue, 07/13/2010 - 13:17 | 466577 playitcool
playitcool's picture

It's called safari ~.^

Tue, 07/13/2010 - 01:04 | 465784 Cistercian
Cistercian's picture

 Great job Tyler...it is always great to be way ahead of the pack.

  Even if it only confirms the dystopian financial slave labor camp I live in......

Tue, 07/13/2010 - 01:14 | 465787 Eric Cartman
Eric Cartman's picture

Wow, compelling shit! Also, just a side note - wtf is up with the ugly bitch on Bloomberg that I have to watch for the next 3 hours? Why can't they keep the hot chick that is on from 12a-1p!? God damn, I like her a lot. I bet if she was given that time slot, she'd drive those Asian markets straight up!

Tue, 07/13/2010 - 01:21 | 465799 vs18
vs18's picture

I will donate constantly if you posted more stuff like this... keep it up Tyler.

Tue, 07/13/2010 - 01:24 | 465800 bob_dabolina
bob_dabolina's picture

Basically, that stock picking course I took as a lad at Eagle Brook is pretty much the largest waste of my time to date. Also, my economics courses from college, my business courses, and...well..ay y not just fuk the speling clas frum fort grade

Tue, 07/13/2010 - 01:27 | 465803 Testicular Cancer
Testicular Cancer's picture

Economic Skynet boys. Someday, it will become self-aware.

Tue, 07/13/2010 - 01:56 | 465830 thesapein
thesapein's picture

If it's not already.

Tue, 07/13/2010 - 01:32 | 465811 Cursive
Cursive's picture

What is most troubling, is that micro similarities, as postulated by non-linear theory, tend to rapidly evolve into massively scaled topological disturbances, and thus a few simple resonant trades can rapidly avalanche into a major market destabilizing event.... such as that seen on May 6.

If HFT is 70% of all NYSE trading activity, then there is an obvious market propping bias (would have said long equity, but HFT doesn't hold overnight or for more than a few seconds for that matter).  What makes the HFT ignore selling volume and buy up the dips?  Or do prop trading desks know how to trick the HFT to do their bidding?

Tue, 07/13/2010 - 03:46 | 465884 gs_runsthiscountry
gs_runsthiscountry's picture

3X Levered Index ETF's - like stuffing a .50 caliber round into a .22.

 

Tue, 07/13/2010 - 01:34 | 465813 Zombies On Toast
Zombies On Toast's picture

Knowing this, the question then becomes can some HFT shop force a blowup, or crash, by applying the correct perturbation at the right time? Is that possible, or not? Yikes!

Tue, 07/13/2010 - 01:44 | 465820 bob_dabolina
bob_dabolina's picture

Or can a collusion of big hedge funds throw off PD's HFT by simultaneously ramping or dumping certain stocks/ETF's?

Tue, 07/13/2010 - 01:45 | 465821 bob_dabolina
bob_dabolina's picture

Or can a collusion of big hedge funds throw off PD's HFT by simultaneously ramping or dumping certain stocks/ETF's?

Tue, 07/13/2010 - 02:02 | 465833 thesapein
thesapein's picture

The answer is already in the pattern, if its fractal. The short answer is yes. If you see what looks like big moves in short time spans, then you can see magnified moves on large times scales. Not good if its more than noise...

Tue, 07/13/2010 - 07:50 | 465949 mephisto
mephisto's picture

Absolutely they can. And if they all have simlar code, they all do similar things at similar times, they may not have to mean it for it to happen. Hence the flash crash. Deliberately causing a crash rather than just getting so incestuously linked together in the order book that one happens anyway - that's the step that (probably) hasnt happened yet.

Never mind. Don't worry. Just by Alcoa. They dig shiny stuff out of the ground and sell it and make no money. Buy them, they're coooool.

Tue, 07/13/2010 - 01:37 | 465815 ThinkAVP
ThinkAVP's picture

THIS IS CRAZY!! Algorithmic trading would be used in any investment strategy.. I guess we are better off using the MAJIC 8 BALL..

 

This still doesn't prove much, in 2008 alone  Black-Box trading brought in roughly 21Billion in profits in 2008, in 2009 high frequency firms account for 73% of all U.S trading VOLUME.. 20% of all options volume are now on this along with Foreign exchange markets..  THE MAJIC 8 BALL, thats prob what we will need or how about that octopus that predicted the world cup all we need is every trading firm to have one and WALLLA $$$$$$$$$$$$$$$$$$$$$$$$$$!!

 

It's ok!! Failure is what makes us.. No need to re-invent the wheel just need to make this $hit better!!

 

 

 

 

Tue, 07/13/2010 - 02:00 | 465832 TraderTimm
TraderTimm's picture

The implications are staggering. It may be that there is a given 'tipping point' that could indeed trigger HFT-cascades. But given the chaotic nature of initial conditions, it would be nearly impossible to solve it in advance to either profit from it or defend yourself.

Looks like we're back in the stone-ages for intraday market forecasting, hoping the HFT lightning doesn't toast our comparatively underfunded accounts in one go.

 

Tue, 07/13/2010 - 02:07 | 465837 thesapein
thesapein's picture

Oh, my. I just realized that your "tipping point" may actually be irrelevant if it is indeed based mostly on fractal geometry, or enough to be significant, because as long as you can see large volatility (variance) in small time scales, then you could see these same spikes on ever larger scales. Oh, my. 

Tue, 07/13/2010 - 02:09 | 465839 thesapein
thesapein's picture

(After two glasses of wine and rereading the paper...)

Hey, everybody. You know what else this means? The stock market could rally or crash no matter the health of the economy. No, really.

Tue, 07/13/2010 - 02:19 | 465846 Caviar Emptor
Caviar Emptor's picture

Wink wink! See my other post

Tue, 07/13/2010 - 02:32 | 465858 thesapein
thesapein's picture

(After taking a few more hits...)

Okay, this is not important if you're a rock, but the implications of this paper are continually surprising me, mostly because I'm now feeling stupid for not seeing it first.

If fractal geometry is applicable to the stock market, then it is only a matter of time before an extreme event will occur. To see this, just imagine any fractal pattern, like say, a sea shore. It may seem to be running North and South, but eventually it makes a significant turn, heading East and West. No matter how much you zoom in or out, this is always the pattern. The question that maybe concerns us most is when does the continental shift occur?

Tue, 07/13/2010 - 02:37 | 465860 Caviar Emptor
Caviar Emptor's picture

...and melding your brilliant insight with Soros reflexivity theory of markets, we can conclude that HFT bots trading with other HFT bots must inevitably result in catastrophic market imbalances. And those tend to implode. Timing it is a matter of not drinking the Kool aid when all the bots are leading the children over the cliff

Tue, 07/13/2010 - 02:43 | 465864 thesapein
thesapein's picture

Nature seems to work that way, doesn't it? I suppose catastrophe has its purpose.

Tue, 07/13/2010 - 02:56 | 465870 Caviar Emptor
Caviar Emptor's picture

+10

Tue, 07/13/2010 - 07:58 | 465953 Troublehoff
Troublehoff's picture

Yep, but nature doesn't 'back out' catastrophe transactions like those exchanges do.

Tue, 07/13/2010 - 05:00 | 465893 fajensen
fajensen's picture

The question that maybe concerns us most is when does the continental shift occur?

No, What actually concerns me is this:

Over time I spend several grands on option bets because I know that eventually the shift will come and the bets will pan out. The magick day arrives - I Am Rich - for a few minutes until my trades are cancelled because "obviously" the pricing was abnormal, a statistical oulier, a loss for Goldman Sacs e.t.c.

So, what the f*ck is the use being in this market?? It's loose, loose always!!

Tue, 07/13/2010 - 06:55 | 465922 drwells
drwells's picture

The amazing thing is that the equity markets haven't been subjected to the run they richly deserve. I'd bet more than a few institutions that have already gone under were more honest and solvent than these fecal Babels.

Tue, 07/13/2010 - 02:44 | 465865 TraderTimm
TraderTimm's picture

The kicker is when all timeframes look exactly the same. Superimposition of a massive correlating wave down or up that reinforces itself via feedback.

Forgive me if you've already stated this in another way already, it is a bit late for me :)

 

Tue, 07/13/2010 - 10:59 | 466221 thesapein
thesapein's picture

Now, I see better why stops are becoming more of a topic, as Tyler already knew. Next though, they'll come up with ways of trading around stops, which means, back to the drawing board.

Tue, 07/13/2010 - 06:20 | 465908 zhandax
zhandax's picture

As Scotty said, "The more they overthink the plumbing, the easier it is to stop up the drain."

Tue, 07/13/2010 - 07:59 | 465954 mephisto
mephisto's picture

Impossible? Not really true - there are guys out there passively waiting for signals that a market is approaching a critical point. Of course you can't predict one exactly, but certain market moves are strong signals for these guys.

It's very smart, still rare, and so far has a good hit ratio. It's been around since about 2006. The academic papers are all there for those with the patience and some maths skills. For those who don't have them, it looks weirdly like Elliot wave "theory". 

Tue, 07/13/2010 - 02:11 | 465834 FreakuentFlyer
FreakuentFlyer's picture

I predict that the average quality of responses to any ZH article has a negative correlation with the number of responses to said article.

I also predict that ZH's ignorance of the previous prediction will be positively corelated to whatever SEC blame is most popular at ZH during the week of the said article.

 

Edit: My "last" prediction is that we will have market crashes in the future, unlike any crash before the given crash.

:)

 

Tue, 07/13/2010 - 11:19 | 466265 thesapein
thesapein's picture

As processing power increases, the more dense the curve over time. An event that may have taken years could happen in minutes. Perhaps, your last prediction would had already occurred had someone not been able to pull the plug on several occasions. That off button is going to get a lot more use. 

Tue, 07/13/2010 - 02:12 | 465838 Caviar Emptor
Caviar Emptor's picture

Super job, Tyler! Impressive 

HFT is the ultimate demonstration of Soros' Reflexivity theory in action. Anticipation of events can make events happen that otherwise wouldn't in self-reinforcing ways that sway the market toward huge imbalances.

Of course the absurdity is even beyond Soros now because HFT is becoming auto-reflexive: HFT bots anticipating and factoring in other HFT bots and their own anticipation of other bots. Hence bots anticipating themselves. Since human stock market participation is now almost non-existant, the stage is set for even more self-reinforcing, untethered HFT reflexivity.  Which, according to Soros, can lead to a state of dangerous imbalance. Think Flash Crash! What a feeling .....coming soon to a theater near you. Also coming soon: Flash Dash! Where the market moves 800 points in a day on biz as usual. 

Tue, 07/13/2010 - 02:40 | 465861 thesapein
thesapein's picture

You mean like a feedback loop? I'd say we're always operating with a standing wave that could turn into a feedback loop.

Tue, 07/13/2010 - 02:49 | 465869 Caviar Emptor
Caviar Emptor's picture

Yeah...a positive feedback. Soros' point is that rather than the market being some objective judge of economics, the opinions and actions of investors in the markets can sway events. So if everybody thinks IBM is going to 900, that fact alone and nothing else can sway the price in a self-reinforcing way...until it doesn't. He used this to explain events in 2007-8 too. 

My point is now HFT bots can reflexively affect the markets (as we all know) but there's a new twist: they affect each other and affect themselves by affecting other bots. So if you tune an algo, you might set it to expect a buy proggie if you place a trade under certain conditions. Then if there's another buy proggie initiated, you could set the algo to buy. If all the algos are set this way by coincidence, you can get 800 points in a day or weirdities in sectors

Tue, 07/13/2010 - 11:30 | 466295 thesapein
thesapein's picture

This is beautiful.

It's something we've come to expect from human bots, leading to bubbles and pops, but now we've got to use game theory for among the robots? It was bound to happen.

Tue, 07/13/2010 - 02:15 | 465843 Bearster
Bearster's picture

How many, and how big, regulatory failures before people see that regulation itself is impractical, unworkable, and immoral?

While it hurts the legitimate businesses, it gives crooks lots of places to hide.  And in any case, it cannot compensate for structural problems elsewhere.  For example: ZIRP, TBTF, trillions of dollars of "liquidity" that is "pumped into" the market, the regime of irredeemable fiat paper currency, moral hazard, SEC/CFTC/FDIC/OCC/OTS, numerous other regulatory agencies, hundreds of thousands of pages of regulation, etc.

In reality, nothing would be improved by forcing the activity described in this paper to happen at a slower rate, and in any case, you can't put the computer genie back in the bottle.

Does anyone believe that today's market could ever go NO OFFER?  If not, why do you think it went NO BID on May 6 (and will again, probably soon)?  Is HFT the root cause of this, or is there another more fundamental cause of a NO BID market?

Tue, 07/13/2010 - 10:34 | 466190 firstdivision
firstdivision's picture

Yes, becuase not regulating worked out so well and deregulation turned crooks into saints.  We did not regulate dervatives when someone noticed that they were getting out of hand far before the crisis.  CA deregulated their energy and that went so well.  Deregulated the banking wall that allowed TBTF.  You obviously fell inlove with Ayn Rand like Greenspan and fail to see reality from the tip of your nose. 

Now if you were bitching about the worthless regulations on the books, I would agree with you.  But to state that all regulation is bad and causes crime and pain.  Well then we are far from agreement.   

Tue, 07/13/2010 - 11:39 | 466314 thesapein
thesapein's picture

Judge Napolitano says that to regulate should be merely to make things regular. Some amount of regularity seems needed. However, the power to regulate is not a power any one person or group should have unless you can trust people to not give in to abusing this power.

Tue, 07/13/2010 - 02:17 | 465844 carbonmutant
carbonmutant's picture

We should be getting close to a wave function collapse...

 

Tue, 07/13/2010 - 02:23 | 465853 Caviar Emptor
Caviar Emptor's picture

Shhh! Don't say "collapse"... They might hear you....

Tue, 07/13/2010 - 11:42 | 466326 thesapein
thesapein's picture

A measurement of the real economy might collapse the wavefunction. Kind of feels like we're in quantum flux.

Tue, 07/13/2010 - 02:41 | 465863 Pooh-Bah
Pooh-Bah's picture

Bring it on!

Tue, 07/13/2010 - 02:58 | 465874 williambanzai7
Tue, 07/13/2010 - 03:13 | 465880 jbeyer
jbeyer's picture

It's funny how zerohedge pitches this as an academic paper.  It is clearly not peer-reviewed.  And the Bouchet-Franklin Institute in Rochester, NY?  It is this guy's "private lab": http://sites.google.com/site/reggiesmithsci/home

Tue, 07/13/2010 - 04:37 | 465889 Chicago bear
Chicago bear's picture

Too little time passed for this to be peer reviewed, like you say. However some tenured professors quote their own papers before publication since that process takes several years. This Institute is not well known in Rochester (if it exists at all) as you say. However, that said, ZH deserves kudos rather than a dig for a) finding the paper b) posting the paper and c) generally being ahead of the rest of the world, including academia, on insights.

Tue, 07/13/2010 - 06:40 | 465916 Tell me lies
Tell me lies's picture

He has a post office box also. I live in Rochester, N.Y and never heard of the place before.

Tue, 07/13/2010 - 07:05 | 465927 Mercury
Mercury's picture

Peer review ain't what it used to be after Climate-gate.

Tue, 07/13/2010 - 11:44 | 466331 thesapein
thesapein's picture

Plus, it's getting peer reviewed by us, right now.

Tue, 07/13/2010 - 08:18 | 465973 mephisto
mephisto's picture

I've just read the whole thing and instinctively I want to defend this.

First the paper is on the arxiv preprint server - in all the physical sciences these days, that's taken to mean that the paper is reasonable. The arxiv has made peer-review almost meaningless, it's too slow a process. I can't emphasise enough, if it's there then it's OK. Not necessarily correct, of course, but OK.

Second, the data in the paper is wide ranging. Someone gave the guy the numbers, that indicates some level of support and trust. The wavelet analysis is a standard technique, I can't tell yet that the analysis is OK but IMO the idea is sound. 

Third, I love the idea of the guy having his own institute and naming it after his heroes and not himself. Kudos for that alone.    

Tue, 07/13/2010 - 13:08 | 466557 jbeyer
jbeyer's picture

The data in the paper is not wide ranging.  It's incredibly easy for anyone connected with an academic economics department to get this data through Wharton WRDS.

"arxiv has made peer-review almost meaningless" ?  Uhhhhh, wrong.  This is so laughable that I don't know where to start with it.

This paper essentially shows that market structure started to change around 2005, which happens to be, incredibly roughly speaking, around the time that HFT started to emerge.  The author doesn't show AT ALL that HFT caused the market structure changes.  One could also show that there is a temporal link between said change in market structure and assets in ETFs.  But correlation does not causation prove.

Tue, 07/13/2010 - 15:39 | 466986 mephisto
mephisto's picture

I have been reading arxiv online since 1992. Nobody in physics reads journals any more. If a paper is amended by a journal referee, then the paper is updated on arxiv and arxiv informs readers of the official journal entry for future references. It's simple, it works, and everybody uses it.

Peer review and journal publication is just a final rubber stamp. That has some people worried, but really it has always been the preprints and talks that matter. Now things just move faster, thats all. Peer review is still used for hiring postgrads, checking productivity for funding requests etc. But all that is just admin. For the subject itself, my statement holds.

I know the data is available to academics. That's the point. He's an academic of some sort, the previous poster seemed to doubt that. 

Of course correlation isn't causation. That's a conjecture. To remove the ETF possibility just choose 2 stocks whose volume is large compared to the indices or ETFs they are part of. Shouldn't be hard. I think the conjecture is a good one, as I can't think of many other possibilities. Can you?

 

Tue, 07/13/2010 - 16:24 | 467100 jbeyer
jbeyer's picture

mephisto, he got an MBA from MIT.  That could easily be how got access to the data.  So the argument that having access to the data somehow validates him as a researcher is total BS.

I wasn't seriously suggesting that the rise of ETFs caused the shift in market microstructure.  I was simply pointing out that the market dynamics are constantly changing, and simply to say that two things happened during similar timeframes is an incredibly weak argument that one caused the other.

Wed, 07/14/2010 - 05:05 | 467756 mephisto
mephisto's picture

Fair enough. The evidence is circumstantial - isn't it always that way in econometric/econphysics? - but to me quite compelling. Maybe that says more about me than about the evidence.

Tue, 07/13/2010 - 08:56 | 466024 Cheeky Bastard
Cheeky Bastard's picture

arXiv.org is as good as it gets when it comes to publishing scientific papers. This paper was published not even a week ago. Also Grisha Perelman published his proof of Poincaré Conjecture on arXiv.org; not some mathematical journal and you do know what an achievement was that proof [hint; biggest mathematical achievement in the last 150 years [OK Wiles and his proof of Fermat's Last Theorem match the importance]].

Trying to invalidate the conclusions [or the article as a whole] simply by evoking "This guy is not from Harvard and/or Princeton] is laughable and mildly moronic on your part. But, sure try to have a go at this article.

Also arXiv.org is where all papers get published sooner or later; some papers may have first been published in journals, some papers may have first been published on arXiv and later submitted to journals; but as the commentator above said; arXiv.org practically made publishing in peer journals obsolete since anyone can pick up a paper, review it and post any observations, counter-argumentation etc etc on arXiv.org and have it peer-reviewed.

Tue, 07/13/2010 - 04:36 | 465890 Privatus
Privatus's picture

Peer review: because a collective of interested insiders necessarily catalyze objectivity.

Tue, 07/13/2010 - 04:43 | 465891 Andrew G
Andrew G's picture

Just like on any other day, Frankie logged onto his computer, took a long sip of his espresso and cracked his fingers. The day before the market was up 2.6% and things were looking even better for today. The pharma stocks he'd bought 3 days ago were also doing nicely and were getting a bit stale and were due for selling.

"Enter stock code", prompted the black and green application.

"FRQR", Frankie typed.

"Stock code invalid", re-enter stock code".

"FRQR"

"Stock code invalid", re-enter stock code".

"FRQR"... "Fuckin keyboard" grumbled Frankie

"Stock code invalid", re-enter stock code".

He paused for a moment. He tried another one of his stocks.

"Stock code invalid", re-enter stock code".

A few more stocks all showed up as invalid. The very confused Frankie clicked on the List Stocks option and got the following message:

"Stocks can only be listed or traded when stock codes are provided by users. Anyone who cannot provide valid stock codes is the stock code"

Frankie fell a tap on the shoulder - 2 men in black uniforms asked him to follow them and silently led him away.

Tue, 07/13/2010 - 06:38 | 465914 buzzsaw99
buzzsaw99's picture

I'm sorry Dave, I can't let you do that. [/HAL9000]

Tue, 07/13/2010 - 06:46 | 465918 Oh regional Indian
Oh regional Indian's picture

Not a math head, so skipped that, but the gist of the paper, like Thespian said up front is self-evident once you see it.

High Frequencies tend to ignore macro factors since the algos are only playing "price" functions, not "value" functions.

Traditional stock-picking was a value game. Trading, of which HFT (anyone notice how close that is to SHTF) is just a logical extension, is a price game.

So, out goes the real world (Value) and in comes mark-to-model (Price).

Price Vs. Value

Simple.

ORI

http://aadivaahan.wordpress.com

Tue, 07/13/2010 - 11:50 | 466345 thesapein
thesapein's picture

lol.

HFT to SHTF, that's great.

 

Tue, 07/13/2010 - 07:08 | 465929 scratch_and_sniff
scratch_and_sniff's picture

 "Correlations previously only seen across hours or days in trading time series are increasingly showing up in the timescales of seconds or minutes."

hummm, thats a bit of a push.

"Second, it is an open questions whether the benefits of liquidity offset the increased volatility."

Its not that there is more liquidity at times, its that it can dissapear in a nanosecond!!! What he giveth with one hand he taketh away with the other(twice as fast). Surely this is not even an argument?

Tue, 07/13/2010 - 07:19 | 465935 Mercury
Mercury's picture

...even the non-member curbside traders outside the coffee houses gradually evolved into over-the-counter (OTC) traders and later, the NASDAQ.

Actually I believe this describe the history of The AMEX too which was referred to as "The Curb" in Street lingo until their business shifted away from stocks.

Tue, 07/13/2010 - 07:37 | 465945 scatterbrains
scatterbrains's picture

Are dark pool transactions reflected in daily volume bars or should technicians throw out all volume related analysis ?

Tue, 07/13/2010 - 17:55 | 467315 New_Meat
New_Meat's picture

Yamada is saying that (forget, but big number, like less than half) of transactions are off the exchange-reported volume.

- Ned

Tue, 07/13/2010 - 07:46 | 465947 Sean7k
Sean7k's picture

Comes down to choices: do you want to trade in the markets or make profits in more traditional areas? Rents, loans, etc.

Do you want to use a computer or go off the grid? 

Each choice has its' benefits and its' limitations.

Technologies will always evolve that benefit their creators at the expense of everyone else. My utility is much more important to me that yours is to me. HFT is no surprise. Excellent work Tyler!

Still, why would we be surprised? If you haven't moved within your own financial fortress of individual design- time is running...

Tue, 07/13/2010 - 07:55 | 465952 Troublehoff
Troublehoff's picture

Now the iZH App is a good idea but having witnessed the hostility to Apple fan boys here I'm not sure how popular it would be.

However, I offer my services free of charge to write said App. Tyler, give me a shout if you're interested. I have a developers licence and a years iPhone dev experience although I'm primarily a web/Java developer.

I fear most ZH'ers are android fanboys as this site really embodies the spirit of open source.

Tue, 07/13/2010 - 09:21 | 466067 Missing_Link
Missing_Link's picture

I'm not sure what is up with the Apple hostility.  It's a great company and makes fantastic products.  I appreciate Tyler's sarcasm and wit but it's a bit misdirected when applied to Apple.

Troublehoff, I'd love an iZH app for my iPhone.  I hope you decide to make it!  You've definitely got a customer right here.

Tue, 07/13/2010 - 14:13 | 466730 playitcool
playitcool's picture

No need for a native app, IMHO. Just write a decent mobile site and the job is done. 

Tue, 07/13/2010 - 08:25 | 465984 Missing_Link
Missing_Link's picture

Terrific article.  I've been waiting a long time to read something like this.

Tue, 07/13/2010 - 08:39 | 466003 Byte Me
Byte Me's picture

Well, that's that then.

What most at ZH already know: buy and hold is gone the way of the dodo.

The efficient matching of capital and risk is subverted by the larceny of High Frequency Thieves draining liquidity from the various markets.

Kiss your pensions goodbye.

Kiss your LT investment capital goodbye.

 

We as traders / investors don't stand a freakin' chance against this HFT lunacy.

It will clearly end badly for everyone. Sure, I suppose algos could be designed to perpetrate flash crashes / mini bull-spikes, but what they will NEVER acomplish is the restoration of public confidence when such events destroy that confidence.

This is one shell game I'm out of. FX has more appeal, it would seem.

Tue, 07/13/2010 - 08:40 | 466006 Waterfallsparkles
Waterfallsparkles's picture

HFT Computers have gotten rid of Day Traders.  They get a stock to a certain pre determined price and hold it there all day churning, churning, churning.

It used to be that a Stock went up in the Morning then sold off, Buyers stepped in and it went up, when it got high enough it went down.  Traders would jump in and out all day long.  A Stock would go up and down a $1. or so every day.  As a Trader you rarely got hurt but with HFT if you have bet in the wrong direction there is no excape except to take a loss.

With the Traders out of the Market it is easier for the HFT Computers to control the Market as no one will Trade anymore.  That is why there are no bids when stocks go down.  The Dominance of the HFT Computers have pushed out Traders and by default they  have become the Market by default.

What I find most interesting is that with no Traders in the Market they have no one to scalp anymore. Except other slower HFT Computers.  As Computers try to out Trade other HFT Computers you end up with Market Crashes.

Kind of like a Canabal not having any outside People to eat so they start to eat their own.

Tue, 07/13/2010 - 08:53 | 466021 Kat
Kat's picture

Simply put:

 

HFT is virtually the only stock trading that is a money maker given the increasing number of SEC trading restrictions and growing regulatory costs.

You guys wanted to punish the market makers and short sellers.  The SEC obliged.  Now, you're not happy with the consequences.  

Deal with it or wave bye-bye to all liquidity.

Tue, 07/13/2010 - 09:01 | 466035 LePetomane
LePetomane's picture

For the time being, it would seem that the wealthiest are content to leave it rather than take it.

 

http://dealbook.blogs.nytimes.com/2010/07/09/for-wealth-managers-10-tril...

Tue, 07/13/2010 - 09:16 | 466061 Quinvarius
Quinvarius's picture

HFT does not provide real liquidity.  HFT quotes are "quantum quotes"  They both exist and do not exist.  If you see them, or measure them, they disappear.  The only way you might hit one is a leap of faith when putting in an order and seeing .01 cents they hit you for.  If one quote gets pulled, there is a "spooky action at a distance" across the market that causes other quotes to also get pulled.

The whole thing is very disturbing.  It will probably create a black hole if we apply enough voltage to the HFT machines.

Tue, 07/13/2010 - 09:39 | 466097 Oh regional Indian
Oh regional Indian's picture

:-)

That was a smile-bringer.

 

ORI

http://aadivaahan.wordpress.com

Tue, 07/13/2010 - 10:25 | 466179 Joe Shmoe
Joe Shmoe's picture

Great article.  Thanks.  I also think system dynamic modeling is a good approach to understanding this problem.  I studied this for a time back in school.  Very interesting stuff.  The quote below (from the above article) is just what system dynamics people examine.    

resulting in volatility surges, recursive feedback loops, and a market structure which is increasingly becoming a product of the actual trading mechanism

The simplest example of system breakdown that I remember was highway traffic.  Highways can handle a great deal of traffic.  Each additional car doesn't seem like such a big deal, until eventually the tipping point is reached.  At the edges of capacity, it might take something tiny, like an area of sun glare or a cop car parked on the side of the road, for the entire system to (in this case) seize.

 


 

Tue, 07/13/2010 - 21:55 | 467628 Veeken
Veeken's picture

acutally the maths is tame...been humbled enough to know.

Sat, 08/14/2010 - 10:54 | 521641 herry
herry's picture

Really this is a great post from an expert and thank you very much for sharing this valuable information with us................ windows vps | cheap vps | cheap hosting | forex vps

Fri, 01/14/2011 - 07:08 | 875568 david22
david22's picture

I just realized that your MB2-632 "tipping point" may actually be irrelevant if it is indeed based mostly on fractal geometry, or enough MB2-633 to be significant, 70-441 because as long as you can see large volatility (variance) in small time scales, then you could see these same spikes on ever larger 70-442 scales.

Thu, 02/24/2011 - 00:34 | 991741 shawnlee
shawnlee's picture

Absolutely they can. And if they all have simlar code, they all do similar things at similar times, they may not have to mean it for it to happen. Hence the flash crash. Deliberately causing a crash rather than just getting so incestuously linked together in the order book that one happens anyway - that's the step that (probably) hasnt happened yet.

Never mind. Don't worry. Just by Alcoa. They dig shiny stuff out of the ground and sell it and make no money. Buy them, they're coooool.
technomarine watches|642-456|650-195|pandora bangle

Do NOT follow this link or you will be banned from the site!