Has it ever happened to you, to build a great product, but have no users?
You know your product is great, but you don't know where to find clients.
You could search on internet, but it'd be too long to find and send emails to every potential user!
Web Scraping could be a solution for you.
Today we’re going to see:
A Web Scraper is a tool used to extract data from websites.
It can have many forms:
Let’s say you want data from Y Combinator's top 100 companies, you can get it by hand in the Startup Directory: But there’s a few problems:
By using a Web Scraper, you can get this data fast and automatically !
A simple Scraper will do the following steps:
All in seconds.
Yes, Web Scraping is legal, technically.
During the Linkedin vs. hiQ labs trial, the US court of appeal ruled that since Linkedin data was public and not copyrighted, there was nothing illegal about scraping it.
Just make sure that your targeted data is public and that you don’t use it illegally, of course.
Here a few links on this subject:
Virtually any piece of public data on the web (and even less public ones.. 👀).
The data you want is just a Google Search away, whether you’re looking for Financial Assets, Demographic data, Song Lyrics or even Movie Scripts, chances are your beloved data is already available online. Same goes for company data, as we’re going to see later.
There are plenty of tools for every type of use and skill level.
Web Extensions like Web Scraper are a good start:
No Code tools like Phantom Buster offer great possibilities:
Keep in mind that more and more websites are protected against Web Scraping. So developing one yourself makes it possible to bypass most security measures.
Let’s say my prospects are French E-Commerce Startup.
I found this website, which is a directory of French Startup, with an e-commerce category:
I've hidden sensitive data, translated the text and modified the page structure for a better understanding of this article.
We can get the following data for each startup of this category (more than 650):
Now we can program a simple bot that retrieves information from each startup and use APIs such as Twilio to send SMS like this:
Such a program can be a powerful Lead Generation tool for your business.
Let me know if you’re interested on a detailed tutorial on how to make a bot like this.
Web Scraping is a powerful and cheap way to get a massive amount of data, legally.
You don’t even need to have IT skills to use one, and many tools allow you to use extracted data automatically.
Sometimes, there are even simpler alternatives to Web Scraping without any need of building something.