Crawling with Scrapy – How to Scrape a Single Page

Web scraping is something that can be really useful, inevitable and a good framework makes it really easy. When working with Python, I like using Scrapy framework because it’s very powerful and easy to use even for a novice and capable of scraping large sites like If you haven’t…


Web Scraping in Python with Beautifulsoup

I’m often asked, “Which web scraping library should I choose?” I usually answer choose the one that is the most popular in your programming language. If it’s java then choose Jsoup. If Python BeautifulSoup is your best bet.   BeautifulSoup Installation You can easily install the most recent version of…

web scraping legal

Is Web Scraping Legal? Top 3 Legal Issues in Web Scraping

Before 2000, web scraping was a gray area in the legal system of US. There was no significant precedent around web scraping. The first time a company was sued for web scraping related activities happened on December 10, 1999, Ebay v. Bidder’s Edge. Bidder’s Edge was an aggregator of auction listings….


Web Scraping in Java with Jsoup

When I was starting out as a programmer and as a web scraper I was addicted to Java. I didn’t care that other languages existed. I was so stubborn that in my hobby projects I literally used Java for everything. I wrote desktop applications, web applications and Web Scrapers in…

Html parsing

The Ultimate Resource Guide to Html Parsers

Html parsing is the backbone of every web scraping software because you need to parse html everytime. I realized that some of you are struggling with finding the right parsing library for your scraping project. This ultimate resource may help you. I gathered the best available html parser libraries in…

Scraping javascript

How to Scrape Javascript in Java

Nowadays the most popular websites have some kind of dynamic elements and they use javascript to display information. Chances are you have to crawl a website full of javascript content. Designing our web scraper, we should look for simple and pure html web pages to fetch data without hassling with…

Install scrapy

How to install Scrapy on Ubuntu 16.04

Today I’m going to show you how to install Scrapy framework in your system in less than 5 minutes. In ubuntu it’s very easy to install packages so it should be the same now. Install Scrapy On Ubuntu The very first step is to make sure that each dependency of…