High Frequency Trading: All You Need To Know

Tyler Durden's picture

In the aftermath of Michael Lewis' book "Flash Boys" there has been a renewed surge in interest in High Frequency Trading. Alas, much of it is conflicted, biased, overly technical or simply wrong. And since we can't assume that all those interested have been followed our 5 year of coverage of a topic that finally has earned its day in the public spotlight, below is a simple summary for everyone.

To be sure, the thinking behind HFT is hardly revolutionary, or even new. Although today HFT is closely associated with high speed computers, HFT is a relative term, describing how market participants use technology to gain information, and act upon it, in advance of the rest of the market. Near the advent of the telescope, market merchants would use telescopes and look out to the sea to determine the cargo hold of incoming merchant ships. If the merchant could determine which goods were soon to arrive on these ships, they could sell off their excess supply in the market before the incoming goods could introduce price competition.

That said, the real proliferation of technology in trading, started in earnest in the 1960s with the arrival of the NASDAQ, the first exchange to heavily use computers.

Ironically, while some form of HFT has been around for a long time, its true "potential" was first revealed in October 1987 with the first whole market flash crash, which resulted from an exponential propagation of program trading, which like right now with HFT, nobody truly understood. And even though some thought that Black Monday would have taught traders and regulators a lesson, it merely accelerated the incursion of computerized and algorihmic trading into regular markets, to such an extent that HFT now accounts for nearly three quarters of all exchange-based trading volume, while dark pools and other "off exchange venues" - or more markets that are not readily accessible to most - account for up to 40% of all total trading by volume up from 16% six years ago.

The rough chronology of algorithmic trading, of which HFT is a subset, is shown in the timeline below.

Over the past decade, following regulatory initiatives aimed at creating competition between trading venues primarily as a result of the overhaul of the National Market System Regulation (or reg NMS), the equities market has fragmented. Liquidity is now dispersed across many lit equity trading venues and dark pools. This complexity, combined with trading venues becoming electronic, has created profit opportunities for technologically sophisticated players. High frequency traders use ultra-high speed connections with trading venues and sophisticated trading algorithms to exploit inefficiencies created by the new market structure and to identify patterns in 3rd parties’ trading that they can use to their own advantage.

For traditional investors, however, these new market conditions are less welcome. Institutional investors find themselves falling behind these new competitors, in large part because the game has changed and because they lack the tools required to effectively compete.

In brief: The role of the human trader has evolved. They must now also understand how various electronic trading methods work, when to use them, and when to be aware of those that may adversely affect their trades.

Market venue competition began with the Alternative Trading System regulation of 1998. This was introduced to provide a framework for competition between trading venues. In 2007 the National Market System regulation extended the framework by requiring traders to access the “best displayed price” available from an automated visible market. These regulations were intended to promote efficient and fair price formation in equities markets. As new venues have successfully competed for trade volume, market liquidity has fragmented across these venues.

Market participants seeking liquidity are required by regulatory obligations to access visible liquidity at the best price, which may require them to incorporate new technologies that can access liquidity fragmented across trading venues. These technologies may include routing technology and algorithms that re-aggregate fragmented liquidity. Dark Pools – trading platforms originally designed to anonymously trade large block orders electronically – began to expand their role and trade smaller orders. This allowed dealers to internalize their flow and institutional investors to hide their block orders from market opportunists.

The use of these technologies can lead to leaking trading information that can be exploited by opportunistic traders. Information is leaked when electronic algorithms reveal patterns in their trading activity. These patterns can be detected by HFTs who then make trades that profit from them. Competition for liquidity has encouraged trading venues to move from the traditional utility model, where each side of a transaction would be charged a fee, to models where the venues charge for technological services, pay participants to provide liquidity and charge participants that remove liquidity. Many trading venues have become technology purveyors.

Broker-dealers have realized that they are often the party paying the trade execution fee, which is used by the venues to pay opportunistic traders a rebate for providing liquidity. To avoid paying these fees and internalise their valuable uninformed active flow, especially from retail customers, broker-dealers have also established dark pools. By internalising their flow or, in many cases, selling it to proprietary trading firms, they can avoid paying the trading fees that the venues charge for removing liquidity from their order books.

The irony is that in their attempt to streamline and simplify the market with Reg ATS and Reg NMS, regulators have created the ultimate hodge podge of trading venues, information leakage nodes, and countless opportunities to frontrun both institutional and retail order blocks.

 

Before we continue, let's take a look at perhaps the most critical and misunderstood concept around, one which HFT advocates are quite happy to (ab)use without really understanding what it means.

There is more: as we explained back in August 2009, the correct term to focus on isn't liquidity, but Implementation Shortfall, also known as Slippage, which is the toll HFTs collect from investors - this is, on average, the cost of spread and frontrunning. Implementation Shortfall (IS) Costs – comprised of 2 pieces: Timing Delay Costs - Any delay cost incurred between the Initial Decision (Open on Day 1) and the Broker Placement Price. Think of this as the cost of Seeking Liquidity; and Market Impact Costs - Price change between the time the Order is placed with the Broker and the eventual trade price. (those curious to learn more about the nuances can do so at this link).

Why is liquidity so critical? Because it goes hand in hand with the concept of the modern exchange, since the measure of consummated liquidity is a key variable in determining the successfulness of any trade venue. It also goes to show why HFTs never operate in a vacuum but in explicit symbiosis with exchanges. It was Zero Hedge who pointed out in 2012 that HFT is a critical component of exchange revenue streams, ranging anywhere between 17% and all the way up to 32%.

 

It is this inextricable link between the venue and the algos that dominate the venue, that has led many to suggest - correctly - that one of the key culprits for HFT proliferation is the dominant exchange business model, known as the Maker-Taker model, in which the liquidity provider is paid (in practical terms it means paying those who provide liquidity with limit orders even it the limit orders are merely "flashed" subpenny orders frontrunning a major order block), while charging liquidity takers (those who take away liquidity with market orders).  This is summarized in the panel below.

No matter the reason, one thing is certain: the use of HFT has exploded.

With the equity markets becoming electronic and prices quoted by the cent (as opposed to the previous eighths of a dollar), the traditional, “manual”, market makers have found it difficult to keep up with the new technologically savvy firms. The playing field has been tilted in favor of HFTs, who use high speed computers, low-latency connectivity and low latency direct data feeds to realize hidden alpha... or as some call it - frontrunning.

HFTs can follow active, passive or hybrid strategies. Passive HFTs employ market making strategies that seek to earn both the bid/offer spread and the rebates paid by trading venues as incentives for posting liquidity. They do this efficiently across many stocks simultaneously by utilizing the full potential of their computer hardware, venue-provided technology and statistical models. This strategy is commonly known as Electronic Liquidity Provision (ELP), or rebate arbitrage.

These ELP strategies can also be signal detectors. For example, when ELP strategies are adversely affected by a price that changes the current bid/ask spread, this may indicate the presence of a large institutional block order. An HFT can then use this information to initiate an active strategy to extract alpha from this new information.

Active HFTs monitor the routing of large orders, noting the sequence in which venues are accessed. Once a large order is detected, the HFT will then trade ahead of it,  anticipating the future market impact that usually accompanies sizable orders. The HFT will close out their position when they believe the large order has finished. The result of this strategy is that the HFT has now profited from the impact of the large order. The concern for the institutional investor, that originally submitted the large order, is that their market impact is amplified by this HFT activity and thus reduces their alpha. The most sophisticated HFTs use machine learning and artificial intelligence techniques to extract alpha from knowledge of market structure and order flow information.

The ubiqituous presence of HFT also means that one of the key considerations when placing an order is "smart order routing" which take into account such concepts as latency arbitrage and order size. This is furhter simplied in the panel below.

Which brings us to the topic of whether all HFT does is simply frontrunning, legal as it may be, and allowing firms like Virtu to post "liquidity providing","trading" profits on 1,237 of 1,238 trading days. The answer - no. At least not explicitly. The full list of HFT strategies, broken down by their impact on various stakeholders is shown below. Again, at least on paper, some strategies are beneficial if mostly to the retail investor. The biggest question, however, is - is there such a thing as a retail investor left at a time when market trading volume has fallen to decade lows, and where HFT now comprises the bulk of lit volume.

And while on paper HFT does provide benefit, the reality is that in practice the consequences of HFT are almost unique negative. Putting aside the ethical implications of whether one views frontrunning as legal or not, the far bigger unintended consequences of HFT is that it has made trading venues inherently far more unstable and prone to sudden and unexplained crashes. Putting aside the best known HFT-induces market crash, the May 2010 "flash crash", more recently the market has suffered several adverse events as a consequence of the new fragmented, for-profit, market  venue environment. In some cases, these events resulted from the unpredictable interaction of trading algorithms; in other cases they were the result of software glitches or overloaded hardware.

KNIGHT CAPITAL LOSS – OVER $450 MILLION + WAVES OF ACCIDENTAL TRADES

A software malfunction from Knight caused waves of accidental trades to NYSE-listed companies. The incident caused losses of over $450 million for Knight. The SEC later launched a formal investigation.

GOLDMAN SACHS – $10S OF MM + TECHNICAL GLITCH IMPACTS OPTIONS

An internal system upgrade resulting in technical glitches impacted options on stocks and ETFs, leading to erroneous trades that were vastly out of line with market prices. Articles suggest that the erroneous options trades could have resulted in losses of $ 10’s of millions. Goldman Sachs stated that it did not face material loss or risk from this problem.

NASDAQ – 3 HOUR TRADING HALT DUE TO CONNECTION ISSUE

Due to a connection issue NASDAQ called a trading halt for more than three hours in order to prevent unfair trading conditions. A software bug erroneously increased data messaging between NASDAQ’s Securities Information Processor and NYSE Arca to beyond double the connection’s capacity. The software flaw also prevented NASDAQ’s internal backup system from functioning properly.

NASDAQ – DATA TRANSFER PROBLEMS FREEZE INDEX FOR 1 HOUR

An error during the transferring of data caused the NASDAQ Composite Index to be frozen for approximately one hour. Some options contracts linked to the indexes were halted, though no stock trading was impacted. NASDAQ officials state that the problem was caused by human error. Although the market suffered no losses, this technical malfunction – the third in two months – raises considerable concerns.

Which brings us to the culmination of 50 years of changing technology, namely the changing investor-broker relationship.

Traditionally, investors spent their efforts seeking alpha and brokers were charged with sourcing liquidity. Liquidity could be sourced via the upstairs market or the stock exchange. The stock exchange operated as a utility that consolidated liquidity. Beyond generating alpha, the only decision for an investor was choosing a broker to execute their trades. Today, investors are still concerned with generating alpha. However, the trading process required to execute their alpha strategies has become more complex. The consolidated utility model has been replaced by a market that is highly fragmented with for-profit venues vigorously competing for liquidity which is provided primarily by HFTs.

This new environment puts brokers in a difficult position. They have a fiduciary responsibility to provide best execution to their clients. This requires them to invest in new technology to source liquidity and defend against HFT strategies. And because many of these venues now pay rebates for liquidity, which is quickly provided by HFTs, brokers are usually left having to pay active take fees to the venue. And at the same time that brokers are incurring these costs, investors are pressuring them to reduce commissions.

These pressures on brokers’ margins are creating conflicts of interest with their clients. By accessing venues with lower trading fees, or attempting passive order routes of their own, brokers can reduce their operating costs. However, these trade routes are not necessarily best for the investors.

Sophisticated investors now demand granular execution information detailing how their order flow was managed by their broker so they can ensure they are receiving the best execution. While brokers provide aggregate performance reports, investors can build a more complete analysis, including broker performance comparison by using more granular information.

Summarized visually - Before:

And After:

* * *

So putting it all together, what is the current state of the market? Ironically, when one strips away all the bells and whistles of modern technology, it all goes back to a concept as old as the first market itself - namely alpha, or outperforming the broader market.

In order to find hidden alpha, it is important to first understand where market participants are with respect to information utilization. The light grey area in the chart below represents the typical institutional investor, playing the role of the “ostrich” or “compiler”, either choosing to ignore the changes around them or to use information only for basic compliance tasks. Most HFTs belong in the light blue “commander” stage; they take command of the information around them and let it guide their business. Taking advantage of the information opportunity, and finding hidden alpha, requires a firm to move up the stages of adaptation.

COMPLEXITY: This measures the sophistication of the use of information in directing action. Whether the information is trade data or newsfeeds, it can be put to use in more or less sophisticated ways, from simple arithmetic to complex statistical methods coupled with strong strategic understanding. Arithmetic uses aim at providing no more than basic accounting measures of values, volumes and gains and losses. Statistical methods aim to identify patterns in information that can be used to guide trading. Strategic understanding introduces game theory, anticipating the reaction of other market participants when an investor employs a particular trade strategy.

FREQUENCY: Each trade an investor makes provides an opportunity to learn. Gathering information from every trade, as opposed to a select few, helps give the investor a better understanding of how those trades may perform in the future. The more frequent the analysis, the more relevant the findings will be.

ITERATION: Findings serve a purpose only if they are acted upon. The key is to use information to guide actions whose outcomes are then analyzed and the findings reapplied. This creates a continuous iterative loop that drives towards ever greater efficiency.

BREADTH: Knowledge sharing with similar objectives (e.g. institutional investors trading large blocks) could lead to a more efficient investment implementation process for all participants. Working together, institutional investors can share block order implementation experience and data, as a utility. The result of this could help participating institutional investors defend against market impact losses and protect proprietary strategies.

HFT firms will likely plateau at stage 4, “commander”, as they are less likely to share any information in a utility concept; trade execution is their proprietary intellectual capital. Institutional investors, on the other hand, have the potential to reach stage 5, “optimizer”. For institutional investors, their proprietary intellectual capital usually lies within their investment decisions, not their trade implementation routes. Institutional investors are thus more willing to collaborate with eachother to work against trade strategies that cause them market impact.

Regardless of an investor’s disposition towards trading strategies; leveraging advanced technology or committing to more traditional trading strategies, it is important to realize that advanced technology trading is today’s reality. Investors need to strongly consider taking the appropriate steps to protect against the potential negative repercussions of, as well as position themselves to find the hidden alpha within, today’s advanced market.

* * *

So the bottom line: HFT is legal frontrunning... but also so much more.  In fact, like the TBTF banks, HFT itself has become so embedded in the topological fabric of modern market structure, that any practical suggestions to eradicate HFT at this point are laughable simply because extricating HFT from a market - which indeed is rigged but not only by HFTs at the micro level, but more importantly by the Federal Reserve and global central banks at the macro - is virtually impossible without a grand systemic reset first. Which is why regulators, legislators and enforcers will huff and puff, and...  end up doing nothing. Because if there is one thing the TBTF systemic participants have, is unlimited leverage to collect as much capital due to being in a position of systematic importance in a market, rigged or otherwise.

Finally, if push comes to shove, and the fate of HFT is threatened, watch out below, because if HFT's presence, glitchy as it may have been, led to the May 2010 flash crash and the subsequently unstable market which has exhibited at least one memorable crash every single month, then the threat of pulling the marginal trader which now accounts for 70% of all stock churn and volume (if certainly not liquidity) would have consequences comparable to the Lehman collapse.

* * *

Finally, for all those still confused by HFT, here is the ultimate simplification.

Source: Oliver Wyman, Hidden Alpha in Equity Trading

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
SmilinJoeFizzion's picture

How are those Japan futures looking?

toys for tits's picture

These companies have to be fools to put billions on the line for this software mania when we've seen that they have problems getting Excel formulas correct.

 

Oh wait, I'm the fool. I forgot they're TBTF.

Drifter's picture

Gosh, it's all so complicated, way above my head.

My silver coins look so nice, so simple, such a bargain, no HFT worry, no counterparty risk, no Tums and sleepless nights.

Buy what Wall Steet hates. Or say they hate. Doesn't matter, it's a bargain.

lordylord's picture

This is the obvious result of capitalism.  Just kidding liberals/RINOs!  This is fascism. 

Oh regional Indian's picture

In this simple (TLDR) piece, I found the last lines disturbing.

Now they are such a big part of the flow that yanking them would kill the market?

Nice, wait till the silent predator becomes a symbyote and then declare it bad.

So, now, when one dies, the whole thing dies.

How convenient.

ori

nmewn's picture

"Gosh, it's all so complicated, way above my head."

lol...somehow I don't believe you.

Thats a compliment by the way ;-)

zerozulu's picture

If you want very  fast click, goto mouse properties and drag the slider to the extreeem right. That all you can do.

Drifter's picture

Yea, it really is nmewn.

Ok it's frontrunning, scalping trades a few pips up to a few cents when trades pass the HFT on their way to the exchange, I get that. And it's just at-market orders not range-bound orders (my terms), I get that. And it's done in a few milliseconds so it's not noticed in time stamps, I get that.

But those charts make my head swim. And I really don't care, I'm not in those markets.

I'm not dumb, show me an electrical schematic, give me a couple minutes with it, and I can tell you what you wana know about it.

I just don't care about this HFT stuff, not even academically, it doesn't affect me.

And it's nothing compared to humongous overt frontrunning Fed lets PDs do and trillions in bailouts and all that.

Btw, how did you come up with that strange handle? I can't even pronounce it.

cdm's picture

my money (silver coin ...) says

 

nmewn == enemy within.

 

 

i want to be on his side

when i grow up.

Problem Is's picture

Willfully ignorant Amerikan... not a compliment by the way... And it does affect you...

nmewn's picture

No, it was a compliment.

MeelionDollerBogus's picture

This part is not academic: seeking local optima in a short time window means having no regard for the massive co-operative interfering pattern which can spike prices many large % in just a few seconds or minutes above where they should be, which rips off investors because prices return a few minutes later with massive losses to traders, or worse, cause flash-crashes which then force exchanges to halt, maybe reverse trades, or possibly let damages sit as they may which can chain-react to crash entire economies. That definitely affects you even with zero trades in the system. It's down -1000 in 5 minutes and if not restored will lead to dow -1500 the next day and if that's not rectified in 30 days it means some significant & damaging % loss of the work-force all across the economy.

eclectic syncretist's picture

And what's the difference between HFT and front-running again?  Oh yeah, HFT is still legal.  Thanks Eric (dick) Holder.

Cattender's picture

NEWSFLASH::: FREE $$$$ FROM THE FED = Stocks going UP... and UP and UP! it's so cool and it NEVER FAILS!!!!

lordylord's picture

Until the middle class loses half of their wealth again and then beg for MORE government.

Kirk2NCC1701's picture

"Japan futures", you ask?  How about "iRobot futures" or "Soylent Green futures"?

Personally, I'm going long on "Guillotines futures" and short on "iBanker futures". 

More than "haircuts" or "close shaves" may be eminent during the Big Correction.  ;-)

achmachat's picture

is it possible that you have an ocular fetish?

blindman's picture

t shirts are important.
moar funny is always true too

Oh regional Indian's picture

Nice one BM. Love the tempo and badass bass on that song :-) And the conga. Nobody uses congas anymore. Or such bold sounding guitars either!

blindman's picture

i've got an interesting link for you. i'll leave it here and someplace more conspicuous. jackson makes great records, no doubt about it, great "session" musicians is a big part of it. jesse ed davis on guitar, him and david lindley often associated with great guitar tracks with jackson browne and others or their own. here is the link .... . https://www.youtube.com/watch?v=q9Me42csgzU Life-Changing Clawhammer Guitar Demo - Steve Baughman

blindman's picture

has been said the "eyes are the window to the soul"
and i guess i may have taken that to heart early on
for better or worser. the sight-vision thing seems to
be a nearly infinite source of potential. there,
it is admitted.

Mr Giggles's picture

So the bubble grows larger and larger, no one but business scalping business, it sails away in to the distance taking what we laughingly call money with it. all that counts is reality. I mean come on billionaire what is that about. theft.

kchrisc's picture

Summary of it all: The whole, and every market, is contolled by W.O.P.R.

"To you want to make a trade?!"

gimme-gimme-gimme's picture

I think they can sum this up really easy:

HFT all you need to know
------------------------------

Piece of shit day traders getting ripped off by even bigger pieces of shit HFT front running scalpers.

Both of these guys are the retailer investors enemy, so let them have at each other. Really good entertainment seeing them trying to pickpocket each other.

Cadavre's picture

Piece of shit day traders getting ripped off by even bigger pieces of shit HFT front running scalpers.

The farking HFT advantage is not so much the aglor engines as it is the fact they trade in a completely different dimension than 1 minute tick yokels  managing  portfolios for John Q. Even that zany E-mini crowd gets a goo ass raping from them divine HFT putas . HFTs are the right had of god, They are the invisible hand, They are the fat finger, and "FLASH CRASHES" are not mistakes, they are an intentional gutting of of allocation traders working one minute tick with stupid stop loss 101 contrived trading models.

Buy or sale at market, well do ya, punk?. The HFT see all them "at market" orders on de queues and milk `em dry like a crank slut milks cocks in a meth lab.

Them HFT shops don't need to go go throgh no farking broker brpker, HFT operators don't seed no farking SEC ticket.  HFT get to sift through the entire farking order queue. They don't pay no farking broker fees or execution fees.

THey can cancel "sniff 1em out" orders, on a dime, and reorder without batting a farking eye!

They don't even need no farking money - the FED prints it up for `em on demand. They can can flush the FX market in a nano, and then price up the ask at next nano.

It's not about transaction fees or taxes - it is all about the dominion acquired through a co-located pipe - and dat all it be!

 

What should the sign be on all equity, FX, futuresm and options exchanges?
"
Abandon hope all ye who enter here"

It's a suckers game, more etucal to bet on the outcome of a dog or cock fight than the price moves in paper that gas no value!

 

Stuck on Zero's picture

All you need to know about HFT: People making billions by performing no useful function for society.  No value added.  Nuff said.

 

booboo's picture

Remember, the worlds largest creature the Blue Whale (not to be confused with the London Whale) gets that way by skimming off one of the worlds smallest.

I Write Code's picture

OUT

STANDING

post!

ghengis86's picture

What, no quote stuffing?

earleflorida's picture

Five Years and still the best consumer analysis the average joe-sixpack can get!

Just come to ZEROHedge and get it?

Tyler gives it out for Free... and yet Tyler gets but a fringy-kind-of-Ht from the world of Finance!?!

Absolutely the Best, Tyler!

Your ship has arrived on the world stage...

Regards ;-)> 

centerline's picture

+1.  If ZH keeps up the critical analysis there will be a time that ignoring ZH will become completely impossible.

Seeking Aphids's picture

Very true....I am seeing increasing references to ZH articles in mainstream media....stuff you can't find anywhere else.......ZH kicks ass. 

Philalethian's picture

United...we ALL Stand...

Divided...we ALL Fall...

Family and world community...will save us ALL!

Grace and gratitude for all you ALL express in the good written works here on ZH.

http://www.youtube.com/watch?v=Wu4oy1IRTh8

 

Bankstein Swissgoldberg's picture

I m here to learn about what will happen in the future. Can't find a better source.

Bearwagon's picture

In order to learn something worthwhile here, you'd better put on appropriate headgear. (Yes, I am talking tin-foil, but to be honest, for best results it should at least be a massive Rhenium-hat.) ;-)

dbTX's picture

All you ndeed to know is that if you are a trader outside a HFT operation, the market is rigged.

Drifter's picture

All you need to know: It's a big rich club and you ain't in it.

(credit George Carlin)

Philalethian's picture

"All you need to know is that if you are a trader outside a HFT operation, the market is rigged."

Shucks, the whole dad dern freaking matrix is rigged.

Who profits from war?

http://www.youtube.com/watch?v=5hfEBupAeo4

 

dbTX's picture

Excuse the fat finger.

Ban KKiller's picture

Long live the free market! Regulations? We don't need them as the free market will sort things out for the best.

Criminals almost always rationalize their reasons for crime. At least I do...

 

davey's picture

I have three businesses and have notified Fidelity that unless they change there order flows I will take away there 401 k and payroll business with us.i hope everyone jumps on board and votes with there feet. PS TD Ameritrade has also been put on notice by me personally. Shame on them both.

Solarman's picture

I'll make it even simpler.  It is insider trading.  No different than a land speculator being tipped off to where the highway interchange is going or the next metro station, and buying up the land to sell to the uninformed for a profit.

 

Just faster, in smaller amounts, and more frequently.

espirit's picture

Moar Simpler:

Matrix Metrics.

RaceToTheBottom's picture

So when liquidity makers can generate profits based on trades, they are no longer providing liquidity, they are generating profits.

Who ever designed that system needs to be shot.

22winmag's picture

Perhaps Vietnam would be kind enough to offer the U.S. some firing squad consultants.

RaceToTheBottom's picture

Filling out the H1B documentation......