Despite the best efforts of the search engines, the majority of the Internet is unsearchable with estimates of this “Unlit” Web as high as 90%. As ConvergEx's Nick Colas notes, some of this content (no one knows how much) is dark for a reason - hosting every form of criminal behavior known to man - but the rest from the increasing interest in anonymous Internet use in light of widely publicized government surveillance.
Among the least well understood emerging themes in technology, Colas points out, is the “Dark Web”, adding that Oscar Wilde famously opined that “All human beings have three lives: public, private and secret.” The existing structure of the Internet handles the first two very well. The Dark Web is, apparently, for the third. The first innovation to move from “Dark” to “Lit” Web is bitcoin, but it certainly won’t be the last.
Via ConvergEx's Nick Colas,
If you are a fan of the movie The Princess Bride, you might recall the character of Dread Pirate Roberts. His was as inherited position, with one man handing down the job to a worthy apprentice when he grew tired of the pirating game. This approach allowed a series of people to benefit from the efforts of many predecessors rather than having to build up their own “Brand” on the high seas.
I am sorry to report that the name Dread Pirate Roberts is now not just a memory from a delightful book and movie, but the nom du guerre of a man accused of running a real life drug website and attempting to arrange several contract killings. His real name is Ross Ulbricht, and these are the particulars of his case:
According to Federal prosecutors, the FBI arrested Ulbricht at the Glen Park branch of the San Francisco Public Library on October 1st. They confiscated his laptop computer, where they noted he was logged into a website call Silk Road as an administrator. This was a popular site for the sale and distribution of illegal drugs and other contraband. The FBI had successfully tracked the operation of the site to Ulbricht, according to court documents related to the case.
In documents found on the seized computer, investigators found a journal, which they claim chronicles Ulbricht’s own development of the site back to its founding in 2010. This included the odd fact that he had grown several kilos of hallucinogenic mushrooms so the site would have something to sell when it went live but before other sellers began to offer their own illegal drugs.
As if running an online portal for illicit drugs wasn’t bad enough, Ulbricht also allegedly tried to arrange six murders-for-hire. The reported targets, all of who are still apparently alive, ranged from blackmailers to fraudulent sellers on the site. These presumably eroded Silk Road’s reputation and user trust.
As of this writing, Ulbricht is being held without bail. The Federal government confiscated 144,000 bitcoins as part of the investigation, worth $122 million as of today. Several press accounts of the case theorize that this stash was only part of Ulbricht’s total holdings and that, if true, would give him access to hundreds of millions of dollars in notional wealth with which to flee the country.
The bitcoin piece of this story got some press, given all the recent interest in the online “Currency”; what got lost in the wash was the presence of the “Dark Web” – a parallel, if much larger Internet, to the one we all use every day. A brief description here:
Google, Yahoo and Bing, among other search engines, only track part of the Internet – essentially the bits that website owners want the public to see. Business owners strive to optimize their sites to appear on pages 1 or 2 of a given search, knowing that most users will not travel farther. Time magazine recently ran a cover article on the “Deep Web”, essentially the Internet which search engines do not reach, and estimated that +90% of online content cannot be found by the typical search engines we all use every day. Wired magazine puts the number at 99%. Either way, most of the Internet is essentially “Dark”.
A large chunk of this “Missing” data must come from its formatting, unfriendly to Google/Yahoo/Bing search algorithms. Just consider all the economic data available through the Federal Reserve’s datasets. Typing “FRED inflation” into Google does get you to the St. Louis Fed’s excellent database of economic indicators, but from there you have to enter exactly what you want. Google, among others, is busy trying to integrate this information; there is a link after the text to a paper describing this effort. Another example of the Dark Web: your financial information at your bank or broker, held behind security firewalls but available to you with an ID and password.
Then there is the part of the Internet that doesn’t want to be found, and where the “Dark Web” means something else. Silk Road is one example, and even though that site is now shuttered there are other places on the Dark Web where users can purchase illegal drugs. From there, it gets a lot worse. There are sites advertising contract killings, illegal pornography, money laundering, and stolen financial information. Some press reports link global terrorism to the Dark Web.
Now, you won’t find the Dark Web on your Explorer, Safari or Firefox browser. To find these sites you’ll need something called Tor, short for “The Onion Router”, or other software which essentially makes you anonymous online. These browsers route your heavily encrypted traffic through a rabbit warren of servers around the world, making it nearly impossible to connect your computer to any individual site you might visit. Tor was actually developed by the U.S. Navy for secure communication, but the software is available for free here: https://www.torproject.org [9]
Secure and anonymous access to the Internet is becoming a growth business, and Tor is one of the hottest tickets to that show. The Google search phrase “Tor search” has tripled in the last year, and the query “Tor” is up 100% over the same period. Like most fashion-forward tech trends, searches for Tor cluster on the coasts, in Oregon, Washington state, California and New York. Since search engine capability doesn’t reach the Dark Web – by design, of course – TorSearch is now available for users, with a reported 130,000 sites listed and something like a tripling of use in the last few weeks, according to media accounts.
There is an obvious tension between a growth opportunity for business and the need for society to regulate and control the illegal use of any technology. A few points here:
Aside from search engine companies, there is not much academic research dedicated to the Dark Web. We spent the better part of day trolling the usual scholarly sites, with little success.
Law enforcement seems poorly equipped to handle the challenge. There was one high profile takedown of a server hosting a range of illegal activities in Ireland over the summer (link below), but its success seems to have been caused by a flaw in the Firefox browser software used by Tor. The flaw has since been fixed. It wasn’t a technical gltich that brought down Silk Road – one of the supposed hitmen contracted for the murder-for-hire plot was an undercover agent.
We did find one consumer product offering related to Tor – something called pogoplug – which directs your web traffic into the anonymous network. It cost $49 and has wifi for your cell phone. The downside: Tor is slower than conventional access since your information transits through more connections than the customary point-to-point process.
All that said, we do have one innovation – bitcoin – that successfully made the transition from Dark to “Lit” web. Silk Road was clearly an early enabler of the online “Currency” but it now has the tacit recognition of everyone from the Federal Reserve to business television. Like the Dark Web, its early appeal was anonymity, but its low cost utility for money transfer should allow it to survive increasing regulatory scrutiny. It won’t be a flawless transition, considering its birth in the primordial ooze of the Dark Web, but it seems well on its way.
Can there be other innovations, developed in the shadows of the Dark Web, which hit the mainstream? It seems inevitable. Revelations of government spying around the world provide the notional demand for anonymous Internet use. Oscar Wilde famously opined that “All human beings have three lives: public, private and secret.” The existing structure of the Internet handles the first two very well. The Dark Web is, apparently, for the third.