8000 Scrapy project · GitHub
[go: up one dir, main page]
Skip to content
@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Pinned Loading

  1. scrapy scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 59.6k 11.2k

  2. scrapyd scrapyd Public

    A service daemon to run Scrapy spiders

    Python 3.1k 576

  3. parsel parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 1.3k 155

  4. w3lib w3lib Public

    Python library of web-related functions

    Python 413 106

  5. protego protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    DIGITAL Command Language 79 29

  6. itemadapter itemadapter Public

    Common interface for data container classes

    Python 68 13

Repositories

Showing 10 of 29 repositories
  • cssselect Public

    CSS Selectors for Python

    scrapy/cssselect’s past year of commit activity
    Python 306 62 18 5 Updated Jan 28, 2026
  • queuelib Public

    Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

    scrapy/queuelib’s past year of commit activity
    Python 291 BSD-3-Clause 55 3 2 Updated Jan 28, 2026
  • parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    scrapy/parsel’s past year of commit activity
    Python 1,306 BSD-3-Clause 155 31 13 Updated Jan 28, 2026
  • w3lib Public

    Python library of web-related functions

    scrapy/w3lib’s past year of commit activity
    Python 413 BSD-3-Clause 106 11 (1 issue needs help) 6 Updated Jan 28, 2026
  • scrapy-lint Public

    A linter for Scrapy projects.

    scrapy/scrapy-lint’s past year of commit activity
    Python 20 MIT 4 42 (2 issues need help) 0 Updated Jan 27, 2026
  • itemloaders Public

    Library to populate items using XPath and CSS with a convenient API

    scrapy/itemloaders’s past year of commit activity
    Python 47 BSD-3-Clause 16 18 (1 issue needs help) 4 Updated Jan 26, 2026
  • protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    scrapy/protego’s past year of commit activity
    DIGITAL Command Language 79 BSD-3-Clause 29 7 (3 issues need help) 0 Updated Jan 26, 2026
  • scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    scrapy/scrapy’s past year of commit activity
    Python 59,577 BSD-3-Clause 11,220 464 (20 issues need help) 204 Updated Jan 23, 2026
  • scrapyd Public

    A service daemon to run Scrapy spiders

    scrapy/scrapyd’s past year of commit activity
    Python 3,090 BSD-3-Clause 576 6 0 Updated Jan 16, 2026
  • itemadapter Public

    Common interface for data container classes

    scrapy/itemadapter’s past year of commit activity
    Python 68 BSD-3-Clause 13 10 3 Updated Jan 8, 2026
View all repositories
0