Boost your Lead Generation with Web Scraping

4 min read -

Has it ever happened to you, to build a great product, but have no users?
You know your product is great, but you don't know where to find clients.
You could search on internet, but it'd be too long to find and send emails to every potential user!

Web Scraping could be a solution for you.

Today we’re going to see:

  • What’s web scraping
  • If it's legal
  • What data you can extract with it
  • Which tools you can use
  • How to Generate Leads

What’s Web Scraping ? 🤔

A Web Scraper is a tool used to extract data from websites.

It can have many forms:

  • An extension driving your navigator, like Web Scraper.
  • A whole program running a “fake” navigator, using tools like Puppeteer.
  • A simple script analyzing the response content of a site with tools like Beautiful Soup.

Let’s say you want data from Y Combinator's top 100 companies, you can get it by hand in the Startup Directory: YCombinator top 100 Companies But there’s a few problems:

  • It’s long: You have to click on each site to get your data.
  • It’s incomplete: You don’t even have emails ! You have to get them yourself, which can make the process even longer.
  • It’s not automated: If new companies come in the top 100, your data becomes deprecated.
  • It's not scalable: the more data you need, the more time you spend.

By using a Web Scraper, you can get this data fast and automatically !
A simple Scraper will do the following steps:

  1. Going on the page.
  2. Getting data (Company name, address, description, logo, ...).
  3. Using APIs such as hunter.io to get emails related to the company.

All in seconds.


Is Web Scraping legal ? 👩‍⚖️

Yes, Web Scraping is legal, technically.

During the Linkedin vs. hiQ labs trial, the US court of appeal ruled that since Linkedin data was public and not copyrighted, there was nothing illegal about scraping it.

Just make sure that your targeted data is public and that you don’t use it illegally, of course.

Here a few links on this subject:


What data can I extract with Web Scraping ? 🎣

Virtually any piece of public data on the web (and even less public ones.. 👀).

The data you want is just a Google Search away, whether you’re looking for Financial Assets, Demographic data, Song Lyrics or even Movie Scripts, chances are your beloved data is already available online. Same goes for company data, as we’re going to see later.


Which tools to use for Web Scraping ? 🛠

There are plenty of tools for every type of use and skill level.
Web Extensions like Web Scraper are a good start:

  • They are easy to use.
  • They can retrieve data from pages with a simple structure.

No Code tools like Phantom Buster offer great possibilities:

  • They automate data extractions.
  • They have many integrations (with tools like Zapier or Integromat).
  • They are highly customizable.
  • They are cheap for basic usage.

If you have basic programming skills, you may want to build a Web Scraper yourself, with libraries such as Puppeteer (Node.JS) or Selenium (Python):

  • The workflow is highly customizable (because you code it yourself).
  • You can browse through the most complex websites.
  • It's free.

Keep in mind that more and more websites are protected against Web Scraping. So developing one yourself makes it possible to bypass most security measures.


Generate Leads with Web Scraping 📈

Let’s say my prospects are French E-Commerce Startup.
I found this website, which is a directory of French Startup, with an e-commerce category: Scraping Usine Digitale

I've hidden sensitive data, translated the text and modified the page structure for a better understanding of this article.

We can get the following data for each startup of this category (more than 650):

  • Name
  • Founder's name
  • Market
  • Creation date
  • Phone number
  • And even more

Now we can program a simple bot that retrieves information from each startup and use APIs such as Twilio to send SMS like this:

Automated cold sms Such a program can be a powerful Lead Generation tool for your business.
Let me know if you’re interested on a detailed tutorial on how to make a bot like this.


Conclusion

Web Scraping is a powerful and cheap way to get a massive amount of data, legally.

You don’t even need to have IT skills to use one, and many tools allow you to use extracted data automatically.

Sometimes, there are even simpler alternatives to Web Scraping without any need of building something.

Please let me know if you found this article useful and tell me what you used Web Scraping for !