In this video, we look at how you can scale out and speed up your web scraping using multiple workers(Spiders) using scrapy-redis.
The article that goes along with this video:
[ Ссылка ]
Scrapy project used in this tutorial:
[ Ссылка ]
Where you can sign up for a free cloud Redis instance:
[ Ссылка ]
Where you can download RedisInsite the Redis desktop manager:
[ Ссылка ]
00:00 - Intro
00:37 - Things to think about when scaling
03:01 - Setting up Redis
05:31 - Adding list of urls to be scraped to Redis queue
08:21 - Scraping with one worker (spider)
12:49 - Scraping with two workers (spiders)
16:59 - Outro
