scrapy multiple sites

How To Scrape Multiple Websites With One Spider

Lately, I’ve come across a scraping job where I needed to scrape the same kind of information from multiple websites. The whole story was to create a spider that scrapes price data of certain products from various ecommerce sites. Also each scraped item needed to have a unique id (uuid)….

scrapy meta

How To Pass Meta Data Inside Scrapy

Not so long ago, I was building a spider which queried product ids from a database before actually scraping the site. The task was to assign specific product ids to scraped products. In the database table I had two columns: product_id and URL. Each URL redirected scrapy to a product…

scrapy mysql

Gathering URLs To Scrape From Database

I have a project where a script dynamically updates a database with URLs the scraper has to scrape. This database contains hundreds of URLs. I had to find a way to fetch all the URLs from the db with scrapy then run the spider on these URLs. Gathering URLs To…

before scraping
sqlalchemy multiple db

Setting Up SqlAlchemy To Use Multiple Databases

In my latest project, PriceMind I wanted to make the database more scalable. I had one database used by Flask. Inside that one db I had the tables a user needs to reach. Also, I had the users table inside this db. But it was dumb because each user has…

spiders quickly

How To Write Scrapy Spiders Quickly And Effectively

This is something new. I’ve just started out the ScrapingAuthority Youtube channel. On this channel you will find videos about web scraping, data processing, data mining, big data and some other stuff. Also, I’m gonna share my progress with PriceMind. As always I appreciate your comments and try to create…

developing pricemind

Building a Web Scraping Based SaaS Business, part 4

Data flow in PriceMind As I talked about it earlier, I’m developing a price intelligence platform for ecommerce companies. If you don’t know, this kind of stuff is heavily relied on data. The most important function I have to focus on is what actionable insights you can get out of…

scrapy spider

How I Write Scrapy Spiders in Minutes

In the last post of my web scraping business blog post series I mentioned that I have a spider-creating system. This system makes me able to build scrapy spiders literally in minutes. With this system my only goal is to be able to produce new spiders for websites as soon…

programming stack

Building a Web Scraping Based SaaS Business, part 3

Hey I’m back again with a new business documentation post. The last time we talked about how I validated my idea without starting to code. Now in this one, I’m gonna go (sort of) deep on the technical side. What programming languages I use for what. Which web framework I…

validate idea

Building a Web Scraping Based SaaS Business, part 2

In the opening post of this series about building my business I gave you a quick overall view what this blog post series will be about. Now in this one I want to be as specific and detailed as possible how I validated the idea before writing any piece of…

1 2 3 4