developing pricemind

Building a Web Scraping Based SaaS Business, part 4

Data flow in PriceMind As I talked about it earlier, I’m developing a price intelligence platform for ecommerce companies. If you don’t know, this kind of stuff is heavily relied on data. The most important function I have to focus on is what actionable insights…

scrapy spider

How I Write Scrapy Spiders in Minutes

In the last post of my web scraping business blog post series I mentioned that I have a spider-creating system. This system makes me able to build scrapy spiders literally in minutes. With this system my only goal is to be able to produce new…

programming stack

Building a Web Scraping Based SaaS Business, part 3

Hey I’m back again with a new business documentation post. The last time we talked about how I validated my idea without starting to code. Now in this one, I’m gonna go (sort of) deep on the technical side. What programming languages I use for…

validate idea
web scraping business

Building a Web Scraping Based SaaS Business, part 1

This is not a tutorial on how to scrape the web. This is something new I’m trying out. I’ve decided to document the whole process of creating my new “business”.  And as you would think from the post’s title it is based on online gathered…

scrapy beautifulsoup
css selector

How to Write the Best XPATH and CSS Selectors for Your Web Scraper

Selectors are one of the most important pieces of your scraper. Well-written selectors make your web scraper work efficiently and fast. When the website’s layout changes your scraper’s selectors need to be changed as well. Then, in a well-established scraping environment the only things that…

web scraping legal

Is Web Scraping Legal? Top 3 Legal Issues in Web Scraping

Before 2000, web scraping was a gray area in the legal system of US. There was no significant precedent around web scraping. The first time a company was sued for web scraping related activities happened on December 10, 1999, Ebay v. Bidder’s Edge. Bidder’s Edge was…

Html parsing

The Ultimate Resource Guide to Html Parsers

Html parsing is the backbone of every web scraping software because you need to parse html everytime. I realized that some of you are struggling with finding the right parsing library for your scraping project. This ultimate resource may help you. I gathered the best…