Scrapy 分布式爬虫框架

使用Scrapy和scrapy-redis进行大规模网络爬取，实现分布式任务队列。集成Splash进行JavaScript渲染，通过scrapy-mongodb管道将结果存储到MongoDB中，并遵守robots.txt规则和AutoThrottle功能。

Scrapy 分布式爬虫框架

Orchestrates large-scale web crawling using Scrapy with scrapy-redis for distributed job queuing. Integrates Splash for JavaScript rendering, stores results in MongoDB via scrapy-mongodb pipeline, and respects robots.txt with AutoThrottle.

Installation

Use the upstream install or setup path that matches your environment:

pip install scrapy

Requirements and caveats from upstream:

:alt: Supported Python Versions
It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_

Basic usage or getting-started notes:

.. code:: bash
And follow the documentation_ to learn how to use it.
.. _documentation: https://docs.scrapy.org/en/latest/
Source: https://github.com/1991513ccie-png/skills
Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst