Guest Post: False Positives & The Limits Of Predictive Analysis

Tyler Durden's picture

Submitted by Charles Hugh-Smith of OfTwoMinds blog,

Analytic systems share system limits with financial markets.

Correspondent Lew G. recently sent me a thought-provoking commentary on the limits of "total information awareness" in terms of any information system's intrinsic rate of generating false positives.

In essence, the rate of false positives limits the effectiveness of any predictive system. The process of attempting to eliminate false positives is inherently one of diminishing return: even with no expense spared, the effort to eliminate false positives runs into boundaries of signal noise and generation of false positives.

To the degree that financial markets are ultimately predictive systems, this suggests a systemic cause of "unexpected" market crashes: signal noise and the intrinsic generation of false positives lead to a false sense of confidence in the system's stability and its ability to predict continued stability.

Here are Lew's comments: 

Resources to deal with reality are inherently limited by that reality.

Information, to the contrary, is inherently infinite, because of the fractal nature of reality.

 

A property of that information reality is that 'meaning' is relative to other items of info, and that any single item can change the interpretation of a big set of facts. E.g., "Muslim, bought pipes, bought gun powder, visits jihadi sites, attends the Mosque weekly, tithes ..." can be completely changed in meaning by a fact such as 'belongs to the Libertarian Party', even 'is a plumber, 'is a target shooting enthusiast'".

 

This will continue to be true no matter how much info the NSA gathers: it will be a small subset of the information needed to answer the question 'possible terrorist?'.

 

Thus NSA's tradeoff of privacy vs security is inconsistent with reality: no matter how much info they gather, no matter how sophisticated their filters, they can never detect terrorists without a false positive rate so high that there will be insufficient resources to follow up on them.

In other words, if the system's lower boundary is one false positive per million, no additional amount of information gathering or predictive analysis will lower that rate of false positive generation to zero.

Why does this matter? It matters because it reveals that large-scale analytic systems are limited by their very nature. It isn't a matter of a lack of political will or funding; there are limits to the practical effectiveness of information gathering and predictive analysis.

Though Lew applied this to the NSA's "total information awareness" program, couldn't it also be applied to other large-scale information gathering and analysis projects such as analyzing financial markets?

This was the conclusion drawn by the father of fractals, Benoit Mandelbrot, in his book The (Mis)Behavior of Markets. As Mandelbrot observed: "When the weather changes, nobody believes the laws of physics have changed. Similarly, I don't believe that when the stock market goes into terrible gyrations its rules have changed."

All this should arouse a sense of humility about our ability to predict events, risks and crashes of one kind or another. In other words, risk cannot be entirely eliminated. Beyond a certain point, we're sacrificing treasure, civil liberties and energy for not just zero gain but negative return, as the treasure squandered on the quixotic quest for zero risk carries a steep opportunity cost: what else could we have accomplished with that treasure, effort and energy?

This entry was drawn from the Musings Reports, which are sent weekly to subscribers and major contributors.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
GMadScientist's picture

You should tell your "friend" Lew not to bogart that joint.

(yeah, you thought it was rain)

Pladizow's picture

"He who predicts the future, lies, even when he tells the truth." - ?????

King_of_simpletons's picture

He who opines is an idiot. We all are.

Dr. Engali's picture

Let me ponder on that statement for a while. I'll let you know what I think later.

King_of_simpletons's picture

I would like to be known as "Anonymous" when that quote becomes part of the vernacular exactly 50 years after my passing.

TheFourthStooge-ing's picture

Made me laugh absurdously.

I would like to be known as "Anonymous" when that quote becomes part of the vernacular exactly 50 years after my passing.

Alas, alas, and even again alas, this quotient becames popular fifty years past, told originally by "an anonymous ZH commenter", attributed rememberings henceforthly being told by "AnAnonymous, ZH commenter".

Blobbing up Chinese citizenism cannot be stopped.

Bear with it.

StychoKiller's picture

Aren't you the person what's always raving about time machines on Easter Island or was that Akak?

TheFourthStooge-ing's picture

Time traveling flying rickshaws, lightning powered as inventionated by US 'american' Ben Franklinism designery pilfered by duplicitous eternal nature of Chinese citizenism, are somehow very something, and a matting thing regardings to sinking of Eastern Island.

akak's picture

Confusifying you are in the doing of akak with Chinese Citizenism poster and thick wiper of the most cheap propapanda bits, AnAnonymous, whose theory is as often heard here that US 'american' citizenism citizens (do not ask for definition of last, will never be revealed) were solely responsible for the downfall of society on Easter Island due to importation of so-called "US 'american' Citizenism" literally centuries before the founding of USA (and not degradation of environment caused by people themselves, living in total isolation from outside world until many decades after their collapse).  Presumably via utilizings of hypothetical magical lightning-powered time-traveling flying rickshaw.

Not making sense?  It is Chinese Citizenism, it is not supposed to.  Best yet.  Part and package of monolizing the insanitation means.  Same song as sung in the off key since 15th,June, 221 BC.

TheFourthStooge-ing's picture

Tells that the equation is only Chinese citizenism citizenish, that is totally devoid of any actual ground.

Can't stop AnAnontards on assessory failings Chinese Citizenism Communautist Party progating for what it is somehow something.

akak's picture

Your wording of the mind has crustified the progrational means in the most mattering way.  Scraping the truth off the Chinese Citizenism goat and upping it to the nth degree.  Salutes are in ordering.

YC2's picture

That Mandelbrot book should be required reading by anyone who is thinking of getting a CFA.  It would only be about 2% more reading, and might cause you to abandon the other 98% anyways.

ArkansasAngie's picture

So ... when dealing in horse manure a few piles of bull shit rise to the surface?

ArkansasAngie's picture

Or is it a case of -- one that all gardens know -- do not spread horse manure in their gardens because you have weeds spouting randomly

DaddyO's picture

The manure has to be composted to kill off the weed seeds, so how does CHS and his correspondent propose to kill the weed seeds?

DaddyO

GMadScientist's picture

The same way compost does: turn up the heat.

GMadScientist's picture

Hedonistic substitution of flat cow-paddies for mushroom duxelles; stay wary, my friends.

highly debtful's picture

Speaking of the NSA, I've got an idea. What if forum members from all over the world commenting on financial blogs would systematically add the words "OPSEC", bankster", "birth certicifate", and "Chelsea Manning is my role model" to each of their messages and then see how the computer servers at the NSA would cope? I'll bet you they would crash within hours, if not minutes. Hell, throw in "Glenn Greenwald" for good measure and their entire headquarters would probably go up in flames instantly. Good, clean fun.


duo's picture

that's a lot of doors to kick down....

Balkan's picture

Fine! I'd add keywords as "pressure cooker", "backpack", "marathon", etc.

MeelionDollerBogus's picture

Until they add new servers for the sole purpose of indexing adding those keywords, along with every other keyword you can think of. They store the data regardless, all you're wondering about there is indexing & priority for further handling.

highly debtful's picture

Shit. I thought my plan was waterproof.  

MeelionDollerBogus's picture

Ya, I know. Sorry for the bad news. I didn't even mention last time about the custom chips, and FPGAs, which can be used to offload from the main CPU's to do the indexing.

hugovanderbubble's picture

Bristol Myers and Eli Lilly will fight to take over THLD (THRESHOLD PHARMA) BID PRICE CLOSE TO 11.40$ 

MeelionDollerBogus's picture

Not that I ever heard. I could miss such a thing but I hear a lot.

NoDebt's picture

"This will be true no matter how much information the NSA gathers:  It will still be a small subset of the information necessary to answer the question 'possible terrorist?'."

Do you think they don't know this?  Do you think that's even the question they are trying to answer?  Do you think that's air you're breathing?

Terrorism was the EXCUSE they used to set it up.  But not the reason.

BobPaulson's picture

Of course.  Effectively,  it is a big experiment. 

On that note, the same algos are being used to game financial data. How well is that working?

DOT's picture

Next time, if you want full credit, you must show your work.

 

GeezerGeek's picture

Why show the work when something is intuitively obvious to the most casual observer?

Tall Tom's picture

Because intuition is not based upon Logic and Rational Thinking.

 

Often that which is intuitive turns out to be the most misleading.

 

I want you to develop your thought process as logical, rational, and orderly.

 

Thus when I teach Math and Science I want to see the damned work. Get it? Otherwise you receive ZERO CREDIT.

 

Do you know how many fucking times that I have had this same damned discussion with my students?

Manthong's picture

Or.. you can just lie to yourself and others to further your own agenda..

 

"..survey of 12,000 people aged 18 and older in eight countries, commissioned by Intel Corporation
..global attitudes towards technology innovation may indicate that young adults are rejecting technology
.. the findings could be more complicated.  (?)
..Nearly 90 percent of young adults questioned in the poll admitted innovations in technology make life easier, but
        about 60 percent said people rely on it too much and that it can be dehumanizing.
.. A different way to read this might be that millennials want technology to do more for them"

WTF????

POLL: Technology can be dehumanizing...
(Drudge)

rtalcott's picture

Nothing exotic about Type I and Type II errors...try to drive the probability of one low the other goes high....a more interesting question is what the distributions look like.

falak pema's picture

god, these guys invent a new type of mustard or ketchup like the french invent cheese.

Why not keep it simple and say the human uncertainty principle makes all analytical predictions unstable as humans can be very irrational when pushed into a corner; aka a crisis; just like when the sky seems the limit and exuberance seems the logical thing to do like the hippies of old singing Age of Aquarius. 

We are not in the bell curve of natural sciences. We are in the fat tailed curve of social sciences. 

Lets go back to literature and create true value like culture and forget about fiat fed dreams of unlimited material growth and never ending wealth creation that do just the CONTRARY.

Economia as its name implies is about scarcity not about unlimited abundance. 

Big Brother's picture

To expound on Mr. Pema's post.  As with a "fat-tail curve", the average and standard deviation is infinite.  So yeah, no predictable outcome- hedge accordingly. 

rtalcott's picture

 As with a "fat-tail curve", the average and standard deviation is infinite.

 

Say what?  I don't think so.

falak pema's picture

well just for argument's sake what does it do to fat tailed birds...look here :

UPLOAD

Manthong's picture

“We are not in the bell curve of natural sciences. We are in the fat tailed curve of social sciences. “

“Economics as its name implies is about scarcity not about unlimited abundance.”

Excellent.

rtalcott's picture

Some reserve the term "fat tail" for distributions where 0 < alpha < 2 (i.e. only in cases with infinite variance).

 

Indeed a special case however no mention of an infinite mean (although with an infinite sigma maybe that follows) ....not sure a distribution with infinite sigma is physically meaningful in finance or anywhere else.

Big Brother's picture

As always, a good article submitted by Mr. Hugh-Smith.

I will be sure to inform the FBI one day when they knock on my door that they've responded to a "False-Positive".

(Note to self- buy pressure-cookers with cash from garage sales)

Tall Tom's picture

Additionaly purchase Grant's Stump Remover from Home Depot with CASH. (Laboratory Grade Potassium Nitrate)

Dr. Engali's picture

I predict that everything will stay as it is until it changes.

DOT's picture

"Nothing" can stay everything else is in motion.

Do you seek to find the position or momentum?

Tall Tom's picture

When dealing in the Macrocosm I do not believe that Planck Unit Accuracy is required in most cases. Are you seeking the 1040 Standard Deviation? Come on, 40 Magnitudes of difference?

 

Are you that serious in identifying all Black Swans in a Macrocosmic Economy?  Good luck. That will take some CPU. You would be better off mining Bitcoin. You will stand to profit somewhat at least.

Kirk2NCC1701's picture

Judging by the JPG on the cover, I'd advise the girl not to get too happy too soon. 

She needs to find out if the "fucker" who got her pregnant will pony up with the consequences.  Metaphorically speaking.

tradewithdave's picture

The Sunstein "nudge" fail? Never!

tradewithdave's picture

The Sunstein "nudge" fail? Never!

Running On Bingo Fuel's picture

The font is like gray or something making it harder to read.

Please resubmit and stop the fuckery with the fonts.