This page has been archived and commenting is disabled.
Algorithmic Control
A modern day adaptation of Frankenstein's monster... ? Computer algorithms are controlling the financial markets. Their "brains" are far bigger, faster and more powerful than any human mind trying to trade along side. And they're playing with more money. They don't mean to, but they do, cause flash crashes and market distortions that their human creators cannot predict or prevent, let alone comprehend. - Ilene
_Irish_Frankenstein.jpg)
Algorithms Take Control of Wall Street
By Felix Salmon and Jon Stokes, Wired Magazine
Last spring, Dow Jones launched a new service called Lexicon, which sends real-time financial news to professional investors. This in itself is not surprising. The company behind The Wall Street Journal and Dow Jones Newswires made its name by publishing the kind of news that moves the stock market. But many of the professional investors subscribing to Lexicon aren’t human—they’re algorithms, the lines of code that govern an increasing amount of global trading activity—and they don’t read news the way humans do. They don’t need their information delivered in the form of a story or even in sentences. They just want data—the hard, actionable information that those words represent.
Lexicon packages the news in a way that its robo-clients can understand. It scans every Dow Jones story in real time, looking for textual clues that might indicate how investors should feel about a stock. It then sends that information in machine-readable form to its algorithmic subscribers, which can parse it further, using the resulting data to inform their own investing decisions. Lexicon has helped automate the process of reading the news, drawing insight from it, and using that information to buy or sell a stock. The machines aren’t there just to crunch numbers anymore; they’re now making the decisions.
That increasingly describes the entire financial system. Over the past decade, algorithmic trading has overtaken the industry. From the single desk of a startup hedge fund to the gilded halls of Goldman Sachs, computer code is now responsible for most of the activity on Wall Street. (By some estimates, computer-aided high-frequency trading now accounts for about 70 percent of total trade volume.) Increasingly, the market’s ups and downs are determined not by traders competing to see who has the best information or sharpest business mind but by algorithms feverishly scanning for faint signals of potential profit.
[...]
These sudden drops are now routine, and it’s often impossible to determine what caused them. But most observers pin the blame on the legions of powerful, superfast trading algorithms—simple instructions that interact to create a market that is incomprehensible to the human mind and impossible to predict.
[...]
But [measures to avoid flash crashes after May 6, 2010] are not ways of controlling the algorithms—they are ways of slowing them down or stopping them for a few minutes. That’s a tacit admission that the system has outgrown the humans that created it.... “Our financial markets have become a largely automated adaptive dynamical system, with feedback,” says Michael Kearns, a computer science professor at the University of Pennsylvania who has built algorithms for various Wall Street firms. “There’s no science I’m aware of that’s up to the task of understanding its potential implications.”
.jpg)
Unregulated capitalism at its finest.
Read the full article here: Algorithms Take Control of Wall Street | Wired Magazine.
HFT pic credit: William Banzai7
»
- ilene's blog
- 7414 reads
- Printer-friendly version
- Send to friend
- advertisements -
HFT traders to Joseph Schmoe and his E-Trade account:
"Heads I win (front running your worthless ass), tails you lose (my lapdogs at SEC invalidate the trade)."
if the markets didn't have real world economic effects it would be different, but they do... - exactly. Also, the massive size of the financial sector makes it such a dominate force in everyone's life.
the guy supporting hft is nutty
if the markets didn't have real world economic effects it would be different, but they do
in addition the purpose of markets is about capitial allocation, not trading to ake money.
this is where our systtem breaks down.
the markets are used little for their real function, and more a system to make money trading.
you have to ask why something exists, why it was created and what was the purpose. then ask how you can best make them function for the purpose they were designed for.
Since I have never invested or traded in any market, I really don't care about this stuff. But it seems as the prices decreased to allow the small investor/trader in the market, the technology took the entry price advantage away from him. If small market people could compete on equal footing with the big players (who lobby to set the rules), then there would be no market. Who then, could the big players fleece? Each other?...yeah, but that is too competitive, better to take life savings away from small players.
That is indeed a basic requirement of any extortion/farming scheme.
Keep the easiest guy to fleece in the position of being fleeced.
So its the computers that believed Ben Bernake when he said the recovery is taking hold and sold all their paper gold? That is too funny...
+2
So in theory, someone just has to hack the Lexicon system, feed it a bunch of negative headlines, then set up an algo of their own to short the market? Flash crash?
...well that explains why the market reacts to only the headlines on a piece of news even when the body of the text refutes the headline....I had been losing sleep over that...it also explains why DJ always puts a positive spin on their headline despite the meat of the article being negative.
For those defending the algo's, let me ask if you would want to live down wind from a nuclear reactor that was managed solely by computer (we will assume the computer is not infected with stuxnet)?
Where can I buy an Algorithm CD to help out this lonely trader....
Let's hope it doesn't get any input from Judith Miller columns...
It is no longer possible for even the most gifted human to beat a machine in games such as chess, etc. Not only will machines do more of our labor, but it appears they will do our thinking too. I would rather own deep blue than bobbie fisher. I would also rather own a superfast black box that has the entire universe of financial data and sophisticated heuristics than peter lynch
I wanna go out with u tu ilene. Even if u were an ugly old troll. Especially if u were an ugly old troll!
Actually I am fairly good looking late thirties and Obama rich. Not a bad catch. U got my email baby. But i wont hold my breath
Dear More Critical T...
You forget the also get to see your trade and scalp the shit out of it regardless of your human instinct and good call on stock x-y-z....
What is broken is true price discovery and a never ending feedback loop up to Dow 36,00 while great can ALSO take the DOW straight from 36,000 to 3,600...good luck.
In defense of HFT is enough to have you shot...hope your algos stop bullets...
Interestingly, Lexicon (et. al) is simply data aggregation and perhaps some cleansing, but the difficulty is tapping the media markets.
Typically, a presence has to be established in each media network carrier's geographic market. The system often consists of paying a cable subscriber a monthly fee to locate equipment in their residence. In turn this equipment (typical server hardware) plugs into the local cable company with a PC tuner and through closed captioning can basically record the text real-time and report back to the mothership.
This is information gold mining, as local affiliate content (think independant affiliates everyone laughs at) is hard to come by in rural areas...
Ilene,
Hey, I never got your number. Anyway, you are pushing my buttons again, going on about terminators. Last I knew, those were really, really bad.
So, anyway, hope we can go out so that maybe you could answer just what the heck it is that allows the HFT to buy everything in sight with 20 minutes to close...and then sell it at close without collapsing the bid?
Good talk.
Cdad
Funny, I always thought the Borg were collectivists.
yeah? so what? There is something called 'technological singularity' which Ray Kurzweil identified long time ago. And it is unavoidable. Resistance is futile. Why is everybody so against it? That's what I want to know.
Because the chaos of reality will deprive the singularity of electricity.
The idea of a superintelligence untethered from the organisms that created it makes good science fiction but it is just that, fiction.
Computers still work with two dimensional binary data programmed by the monkeys at the typewriters.
To imagine that the binary number crunching calculator will get up and walk away from it's organic progenitors is the narcissistic fantasy of nihilists.
Make no mistake, the machines are the guns and tanks and nuclear bombs of the modern age but they are wielded as always by humans bent on depriving other humans of life, liberty, and the pursuit of happiness (and their assets).
The uniform involves no medals or jack boots or insignia but rather a wool suit, a tie, a sheepskin, and often a silver spoon.
Tisk, tisk, tisk...don't those pretty boys know that their games always lead to the self-absorbed bullies taking over, spoling their party, and tearing everything down via death and destruction?
All hail Caesar.
And why not? Traders trade with the aim of squeezing people from using resources. Traders pocketed the money freed by the move.
So how is it unfair for anyone to work on squeezing traders from using the resources?
Will a GAI perform a better work than traders? Probably not. Worse? neither.
Squeezing out people who pocket bonuses, no matter their results, will free more resources for the guys who implement the GAI algorithms.
GAI not being self conscious, they wont reclaim bonuses.
So what? Do you have anything against this save the petty attempt that tyranny is good until your are tyrannized?
Machines evolve at astronomical rates compared to the fleshlings. Eventually *something* will emerge from that - and maybe it already has; since the 1980's government policies have shifted from supporting wages i.e. People, to supporting assets, aka Things. People are "ressources" now, something to be in the most profitable way.
Perhaps some really smart sucker in the dungeouns of Goldman Sachs accidentially created a Machine God, which is now making GS rich and powerful - of course in return for a few favours here & there ;-). If someone suddently cracks protein folding and acquires molecular nanotechnology then I would worry - Something would be looking to build a Body!
There is so little scientific understanding of what intelligence is and how conscience is created that we are like cave-men with access to a liberal supply of U235: We keep piling those funny rocks up in the bottom of the cave and not much happens until there is a criticality event. With no understanding, no-one will know what just happened.
If we are not the smartest things around, then we probably will never know that we are being manipulated the same way a dog does not know he is being trained.
If we are the smartest things possible in the universe ...well... that is even worse, somehow.
Is this when Skynet takes over?
Forget John Galt: who is John Connor? And where's Sarah? I guess we're not supposed to know ....
price fixing? price fixed? price fix = higher price... if it's broke (dipping), fix it!
Unregulated capitalism fascism at it's finest.
There, fixed it for you.
So I imagine Lexicon's CNBC feed is input to a summing positive feedback loop.
mustard seeds > green shoots > retail investors and traders buy (*HFT machine front runs each order* = free money from any buyer) > market goes up > "recovery" talk continues > perception = 99% of a lie > markets up <market close (sort of)> HFT machines sell next morning or afternoon (depends on # of voltage drop martini's and bandwidth hangovers) and makes more money front running sales) > retail investors and traders "buy the fuc*ing dip" (BTFD) > HFT machines front run sales and make more $ and + FED purchases = levitating market levels on low volume = mustard seeds > green shoots > repeat
+1
That's the obvious danger, and 70% is too high a proportion to entrust to programs designed to act instantly to cumulative effects. A program is only as good as the programmer, and expert systems are a VERY long way away from discerning wisdom from data (For that matter, so are most people). That's why we have armies of people still pre-processing data before it's used.
Someone should do an article on HF Micro trades.
IF: price(x)
THEN: btfd
ELSE: btfd
END
Here is an absolutely constructive question to all the "Danger, HFT causes more volatility which is EVIL!!!" types of people here on ZH:
I see and experience volatility as a bonus to value investors. If I think that an instrument will go up longer term, I can buy the dip - and I can buy deeper dips due volatility.
I.e. a human trader is at a marked advantage, while the robots trading against each other in the volatility space do a zero-sum game of trying to out-penny each other. They take money from other robots. The human trader, who 20 years ago had to pay a spread+costs of 50 cents per share or more, can now do it below 10 cents, even with HFT/robot overhead accounted.
I.e. while HFT may increase volatility, it also provides real liquidity and better points of entry to human traders like myself. So I see them as an unconditionally good thing. (Beyond the fact that I can use automated tools myself to time the market more efficiently - which is a significant advantage as well.)
So exactly what is the harm to human trader here? Honest question, because I don't see any harm to myself, neither in theory, nor in practice - I only see and experience advantages.
I see the only harm to the traditional Wall Street market maker monopolies who cannot seek as much rent anymore - but I'm not shedding any tears for them.
This is why I think the routine bashing of HFT by ZH is fundamentally shortsighted - unless ZH wants to be an anti-HFT front for the banks :-)
In reality HFT punishes the lazy and allows the nimble little guy get a good price too.
Machine might inconvenience the former swarm of parasitic stock-scalpers. I have no pity for them, they'll go the way of the buggy whip makers. Get a life and use your brain for something non-mechanic.
But to the real human trader, machines are a big boon: both as a tool, and as an effect on liquidity and on volatility. FYI 2010 May 6 was an excellent day for human traders to enter the S&P via limit orders.
Unless someone can robustly defend all this HFT-bashing, ZH's stance on HFT is little more than petty luddism perhaps driven by paleo human fear from machines ...
Keep thinking that way, the machines need you to buy.
HFT provides real liquidity? Is this like Mortgage Backed Securities providing capital and stabilizing the mortgage and banking industry?
"Paleo human fear from machines" you say? Oh dear, the "monkey afraid of fire" meme. No, monkey not afraid, monkey very angry at bad monkeys.
This comparison does not hold up to scrutiny. The mortgage market is (and will, for a considerable amount of time) be very illiquid with high spreads and a thin market. If you try to trade it you'll see that vast areas of the market are simply not moving (and not even accessible to the regular 'human trader').
The electronic markets on the other hand are very, very accessible and very liquid if you decide to get in or if you decide to get out in a smart way. For example I just executed a few trades a couple of minutes ago with very low costs. It is real.
As a human trader I simply see no problems in being in the same pool with machines neither in practice nor in theory, and I lived through May 6 and other events of volatility.
The 'machines' you are angry at are really just extensions of the thoughts of other humans, expressed as computer algorithms. If you have a superior idea about where the markets are heading any volatility just helps you get a cheaper price to buy at and a more expensive price to sell at - even with very simple, non-algorithmic tools such as manually placed limit orders.
Machines might be able to beat other machines via their endless weapon's race on small timeframes, but as a side-effect they will also be giving superior points of entry and exit to you the independent, intelligent human trader.
You should celebrate that, not fear it.
Go on pardner, Make Their Day:
http://news.slashdot.org/story/10/10/15/0139217/Norwegian-Day-Traders-Co...
Thy shall not conspire against thy silcon masters.
That was a weird ruling by the Norwegian court (and pretty much the only such ruling in the past 10 years of computerized trading) but note that even in that admittedly weird case those 'human traders' did not use volatility to enhance their fundamental entries and they did not do "human trading" at all.
What they did was that they figured out how an algorithm run by a big market participant was working and wrote their own algorithm to price-stuff it to make big losing moves in rare, illiquid instruments. They repeated this many, many times, using the big-player algorithm as a cash machine, until they got charged.
But if you look at trading volumes even in Norway that are not challenged you'll see that even there this is a very rare exception.
I personally think that those who make big losing moves should be punished on the markets fair and square (be that humans or algorithms) - and that is predominantly so in non-Norwegian markets.
I think so too.
But the losers happens to own the ears of government, which means that they own the courts and all the taxes collected, Denmark f.ex. has 6 years taxes on the line in it's bailout package, Norway is probably in the same pickle. Which sadly means that Not Even Automation killing the wages of finance like it does everywhere else will get rid of the b*sterds. Only total collapse of the Euro will.
All the data I'm seeing says that after-the-fact trade invalidations are exceedingly rare.
If you disagree you need to prove that a substantial portion of trades are being invalidated for the detriment of small-time investors, not just assert it.
Here's something for you to ponder. The only purpose of capital markets is the most efficient distribution of idle capital accumulated via previous work and past savings. Large numbers of entities with independent opinions (i.e. free markets) are the best way to do this compared the alternative of centralized/statist control of capital by "experts". Just as the most efficient engine is that which loses the least amount of kinetic energy to friction the most efficient capital distribution (i.e. financial) system is one that minimizes capital losses due to financial intermediation. HFT is by definition a zero sum game whose frictional losses are proportional to rate of trading (think about it does your broker allow you to trade for free?). So, for example, if I believe (and I do) that there are will be major shortages in food/energy in the future I will invest in companies like POT, BG, DE, BEXP, NOG etc. The capital I provide allows these companies extra funds to invest via equity as well as to finance dept at lower rates (than say Greece). If the companies squander it on executive bonuses or jets I deserve to lose my bet. If I feel (and remember markets are just trading opinions) that banks are basically insolvent, housing recovery is a farce, and gold, platinum and silver are underpriced while supply will have trouble meeting future demand then I will do a pairs trade, shorting banks and using that capital to buy equity in miners who then will develop better, more efficient extraction methods and increase supply accordingly. Anything that distorts this capital allocation process is friction and a waste of financial "energy". HFT fills this criteria to a T.
And if this weren't bad enough, the more complexity you introduce into a non-linear system with multiple feedback loops (e.g. the financial system) the better chance you have of complete chaos. Would you rather fly on an airplane with a jet engine with 400 or 4 moving parts? Would you rather have that engine made in the US or China-then allocate you capital accordingly. Our society is headed for complete chaos as entropy is guaranteed to win in thte long run. HFT is just a simple example of speeding up this process!
QED
Nice, fair and square. But how do you provide for the first option?
So far, what can be provided is a number of entities with interdependent opinions, which are going to hedge against their own bets by peeping over the shoulders of the others.
That is nice, fair and square. But what about optimization limit? Developping better, more efficient extraction methods? Do you have any point to show that this solution is perpetually available? What if miners are nearing optimization limit, which signals a diminishing return on an investment aiming at better extraction methods?
Nice reply, and I agree with some key points of it, except this crutial argument which appears to be at the heart of your argument:
Could one not argue that 'extra noise' (which HFT really is) will increase the efficiency of the markets, because it will reward those like you who have the right kind of long-term ideas about POT, BG, DE, BEXP, NOG, etc. regardless of the micro-action on the markets?
You are given an advantage because the 'noise' allows you to enter lower and exit higher. Since it's all a zero-sum game those monies you get come from traders or algos that are just speculating with no real idea about where things are heading.
(This works as long as price is not 100% noise - and we are very, very far away from the markets being noise alone.)
Lets see a concrete example of how noise works in your favor, even if you yourself do not use computer algorithm to time the markets:
Say brent crude oil trades at $97 today, and there's +- $5 of weekly volatility in its price. Markets are computerized an highly liquid, cost per trade is $0.1. HFTs are out in full force and may scalp off a few cents - but lets be generous and set their cost at $0.05 per trade.
You as the intelligent human trader have the idea that invevitably oil must go to $110 a barrel due to various economic pressures, within a couple of months.
The weekly volatility range allows you to scale into a long position with an average price of say $93. It will also allow you to exit at a $115 top - giving you +$22, with a transaction cost of $0.2 and 'HFT cost' of $0.1 - so you'd net +$21.7. Even if you did not use the range of volatility fully, you could easily have made around $20 with a dollar off both the bottom and the top.
Lets fast forward back 20-30 years.
You have the same idea, the fundamental prices are the same, but the markets are neither as volatile nor as liquid, transaction costs are at $0.5 per trade.
You'd be able to enter your trade at $96 and you'd be able to exit it at $110, so you made +$14 with a transaction cost of $1.0 (which went to the market maker monopoly - i.e. big banks on Wall Street) - so you are netting only $13 with the same fundamental idea - more than 60% less profit!
Furthermore, it is true that the 'HFT cost' was $0.0 back then.
But there were other costs beyond the much more generous comission to your Wall Street broker:
Firstly, you'd have to work the phones (no fancy computers) so you wouldnt be able to 'see the flow' and you'd have a less efficient market entry and you'd be scalped by floor traders $0.5 who were doing that all day. That would be another 4% off your profit.
(Assuming your broker was honest and did not tip off buddies or enter his own trades before yours.)
So I think the critique of HFT (which really just points to the cent-per-share cost that HFT poses to day-traders) fails to consider two much stronger effects: how the volatility and liquidity can be used to your benefit, and how markets really operated 'back in the days' when computerization (at least towards clients) was not frequent yet.
To sum it up, computers mostly took up jobs that only computers should be doing really: just like there's no real place for the buggy whip maker anymore there's no place for human price-scalpers.
But if you have an edge in the market the effects of computers helps you, big time - even if you are a 'pure human trader'.
If on the other hand you try to act like a computer, with no fundamental edge at all, just speculaing all day in and out, you will likely lose against the computers.
And IMO that is a profoundly good thing both in the short and long run.
One harm is that in response to the machine-generated crashes the NYSE and SEC have decided it is A-OK for them to go in after the day closes and decide to invalidate trades. "It was a bad quote." You, the smart small guy, laid your trap, the machines fell-in, you caught them, and your trade will get cancelled after the close. You may have made three other trades to hedge your trapped quarry, or use your profits - that's your problem. Now I'm betting JP Morgan never gets burned by this - but you may well.
That is a valid concern, and if it gets common I agree that it would be bad.
But right now the numbers show that trade invalidation is very infrequent. On May 6 stock exchange officials indeed invalidated a few blocks of trades in a handful of extremely-affected instruments. But, and please point to contrary information if you have it, the overwhelming volume of trades on that day held up and you could make a bunch of money in almost every instrument if you were in the wings waiting.
The complaints here on ZH about "volatility spikes" and "mini-flash-crashes" are an almost daily occurance, but I don't see the NYSE or the NASDAQ invalidating trades on a daily basis.
It only takes one. If it's yours.
So what? People like myself make use of volatility and have been doing so for years and have not had a single invalidated trade so far. I traded VXX on that day and that trade did not get invalidated.
But yes, if you trade only a single time in your whole life and that trade gets invalidated, it's a big deal.
Otherwise, if you are sane and diversified just a little bit then the very low probability of a single trade getting eliminated does not matter much, even if it happens to hit you.
You'd have to prove that a substantial portion of say NASDAQ trades get cancelled.
But if you look at the facts you'll see that in practice only a very, very small propotion of trades ever gets cancelled.
The 'mini flash crashes' that ZH has been reporting in the past couple of months did not trigger a single cancelled trade by market authorities AFAIK. Markets can cope and human traders can use volatility to their advantage.
"You'd have to prove that a SUBSTANTIAL PORTION (emphasis added) of say NASDAQ trades get cancelled.
I admire your optimism that as long as only 15-25% of trades get cancelled, the market will continue to draw suckers. You're that guy who in Auschwitz would have said "hey guys, buck up! The Allies are BOUND to get here eventually!"
No, my point is actually that less than 0.01% of all trades ever get invalidated - and even those do not get invalidated on a daily basis. Trade cancellations are very rare.
No 'faith' is involved - these are public exchanges and the fact when trades get invalidated is public. You can look it up yourself.
I.e. if you think that trades get cancelled on a massive proportion (and even 0.1% would be absolutely horrendously high for an exchange, let alone the 10% figure you cite), you need to prove it, not just assert it.
I see you do have a chance to stand against the machine. I bet your small core i7 extreme edition with latency more than 1k helps you to pick up the right price.