HOW (AND WHY) HEDGE FUNDS ARE USING WEB SCRAPING

There is a growing trend of more and more hedge fund managers turning to the web to procure investment data. And one of the primary ways they’re sifting through it all is with web scraping.

Experts estimate that in 2018, web scraping accounted for 206 billion page views per day, which equates to about 5% of all internet traffic. Here’s a look at the current state of web scraping and the impact it can have on hedge funds.

HOW (AND WHY) HEDGE FUNDS ARE USING WEB SCRAPING

An enormous amount of data is generated on the web – about 2.5 quintillion bytes of data every day. Tapping into that information provides hedge fund managers with the valuable insights they need to make smarter, more lucrative investment decisions.

However, it’s extremely difficult to filter through the massive volume of data that’s generated. Web scraping, a technique that efficiently pulls data from a wide range of sources, streamlines this process and supplies hedge fund managers with the key insights needed to guide investment decisions.

And a growing number of firms are taking it one step further by using full-scale web data integration (WDI), which is similar to web scraping but more robust. WDI normalizes data and presents it in a way that’s easy to consume with reports, visualizations, and dashboards.

For example, investment firms use web data integration to aggregate news articles from across the web for predictive analysis. They then take that aggregated data and feed it into their own machine learning algorithms to guide their decision-making and develop smart investment strategies for their clients.

THE RISE OF WEB DATA IN HEDGE FUND DECISION-MAKING

Having access to key information has always been critical to achieving success as an investor. But “the spread of big data has made it easier than ever to find value in the data hidden in plain sight,” writes CNBC reporter Eric Chemi:

“From companies’ internal analytics to satellite photos, scraped website data to tracking consumers’ locations, hedge funds and institutional investors are always looking for creative ways to get the edge in investments.”

For example, investors use web scraping to quickly analyze millions of tweets to determine which stocks to buy and sell. A single tweet from Hillary Clinton criticizing private prisons in 2015 resulted in the stocks of two top corrections companies dropping by 6% and 4%, respectively.

Another tweet that same year by investor and Los Angeles Clippers owner Steve Ballmer praising Twitter’s innovation resulted in a 5% increase in Twitter’s shares. This illustrates how leveraging data like this, especially at scale, can greatly improve the decision-making of hedge fund investors.

HOW MUCH ARE HEDGE FUNDS SPENDING?

As you might imagine, investment firms are willing to put up a good chunk of money for access to this data. In fact, “Hedge funds will spend $2 billion on web scraping software to gain an edge, and it’s part of an investing gold rush,” Business Insider writes. But when you look at the long-term payoff, it’s often a wise investment and can actually lead to cost reductions for monitoring websites.

For instance, Fiscal Note, a privately-held software, data, and media company, decided to use a web data integration solution to collect a large volume of data from across the web. Before implementing the platform, they monitored 1,000 sites, which was a meticulous process with a high manual error rate. But after implementation, they were able to monitor over 3,000 sites and greatly extend their reach, while at the same time increasing their data quality.

What’s really interesting is that Fiscal Note spent the same amount of money after implementing web data integration as they did before, a 66% reduction in costs (considering they were able to monitor triple the number of websites). Though web scraping comes at a cost, this illustrates the long-term savings it can offer.

FINAL REMARKS