Scrapy仅爬取站点的前5页。

huangapple go评论68阅读模式
英文:

Scrapy Crawl only first 5 pages of the site

问题

I am working on the solution to the following problem, My boss wants from me to create a CrawlSpider in Scrapy to scrape the article details like title, description and paginate only the first 5 pages.

我正在解决以下问题,我的老板希望我创建一个Scrapy中的CrawlSpider来抓取文章的详细信息,如标题描述,并且只翻页前5页。

I created a CrawlSpider but it is paginating from all the pages, How can I restrict the CrawlSpider to paginate only the first latest 5 pages?

我创建了一个CrawlSpider,但它正在从所有页面翻页,如何限制CrawlSpider只翻页最新的前5页?

The site article listing page markup that opens when we click on pagination next link:

网站文章列表页的标记,当我们点击翻页的下一页链接时打开:

Listing page markup:

列表页面标记

    <div class="list">
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-1">Article 1</a>
        </h2>
      </div>
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-2">Article 2</a>
        </h2>
      </div>
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-3">Article 3</a>
        </h2>
      </div>
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-4">Article 4</a>
        </h2>
      </div>
    </div>
    <ul class="pagination">
      <li class="next">
        <a href="https://www.example.com?page=2&keywords=&from=&topic=&year=&type="> Next </a>
      </li>
    </ul>

For this, I am using Rule object with restrict_xpaths argument to get all the article links, and for the follow I am executing parse_item class method that will get the article title and description from the meta tags.

为此,我使用Rule对象和restrict_xpaths参数来获取所有文章链接,然后在跟踪时执行parse_item类方法,该方法将从meta标签中获取文章的标题描述

Rule(LinkExtractor(restrict_xpaths='//div[contains(@class, "snippet-content")]/h2/a'), callback="parse_item",
follow=True)

Detail page markup:

详细页面标记:

    <meta property="og:title" content="Article Title">
    <meta property="og:description" content="Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularized in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.">

After this, I have added another Rule object to handle pagination CrawlSpider will use the following link to open other listing page and do the same procedure again and again.

在此之后,我添加了另一个Rule对象来处理翻页,CrawlSpider将使用以下链接打开其他列表页面并一遍又一遍地执行相同的过程。

Rule(LinkExtractor(restrict_xpaths='//ul[@class="pagination&quot]/li[@class="next&quot]/a'))

This is my CrawlSpider code:

这是我的CrawlSpider代码:

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
import w3lib.html

class ExampleSpider(CrawlSpider):
name = "example"
allowed_domains = ["example.com"]
start_urls = ["https://www.example.com/"]
custom_settings = {
'FEED_URI': 'articles.json',
'FEED_FORMAT': 'json'
}
total = 0

rules = (
    # Get the list of all articles on the one page and follow these links
    Rule(LinkExtractor(restrict_xpaths='//div[contains(@class, "snippet-content")]/h2/a'), callback="parse_item",
         follow=True),
    # After that get pagination next link get href and follow it, repeat the cycle
    Rule(LinkExtractor(restrict_xpaths='//ul[@class="pagination&quot]/li[@class="next&quot]/a'))
)

def parse_item(self, response):
    self.total = self.total + 1
    title = response.xpath('//meta[@property="og:title"]/@content').get() or ""
    description = w3lib.html.remove_tags(response.xpath('//meta[@property="og:description&quot]/@content').get()) or ""
   
    return {
        'id': self.total,
        'title': title,
        'description': description
    }

Is there a way we can restrict the crawler to crawl only the first 5 pages?

有办法限制爬虫只翻页前5页吗?

英文:

I am working on the solution to the following problem, My boss wants from me to create a CrawlSpider in Scrapy to scrape the article details like title, description and paginate only the first 5 pages.

I created a CrawlSpider but it is paginating from all the pages, How can I restrict the CrawlSpider to paginate only the first latest 5 pages?

The site article listing page markup that opens when we click on pagination next link:

Listing page markup:

    <div class="list">
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-1">Article 1</a>
        </h2>
      </div>
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-2">Article 2</a>
        </h2>
      </div>
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-3">Article 3</a>
        </h2>
      </div>
      <div class="snippet-content">
        <h2>
          <a href="https://example.com/article-4">Article 4</a>
        </h2>
      </div>
    </div>
    <ul class="pagination">
      <li class="next">
        <a href="https://www.example.com?page=2&keywords=&from=&topic=&year=&type="> Next </a>
      </li>
    </ul>

For this, I am using Rule object with restrict_xpaths argument to get all the article links, and for the follow I am executing parse_item class method that will get the article title and description from the meta tags.

Rule(LinkExtractor(restrict_xpaths='//div[contains(@class, "snippet-content")]/h2/a'), callback="parse_item",
             follow=True)

Detail page markup:

<meta property="og:title" content="Article Title">
<meta property="og:description" content="Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.">

After this, I have added another Rule object to handle pagination CrawlSpider will use the following link to open other listing page and do the same procedure again and again.

Rule(LinkExtractor(restrict_xpaths='//ul[@class="pagination"]/li[@class="next"]/a'))

This is my CrawlSpider code:

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
import w3lib.html


class ExampleSpider(CrawlSpider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = ["https://www.example.com/"]
    custom_settings = {
        'FEED_URI': 'articles.json',
        'FEED_FORMAT': 'json'
    }
    total = 0

   
    rules = (
        # Get the list of all articles on the one page and follow these links
        Rule(LinkExtractor(restrict_xpaths='//div[contains(@class, "snippet-content")]/h2/a'), callback="parse_item",
             follow=True),
        # After that get pagination next link get href and follow it, repeat the cycle
        Rule(LinkExtractor(restrict_xpaths='//ul[@class="pagination"]/li[@class="next"]/a'))
    )

    def parse_item(self, response):
        self.total = self.total + 1
        title = response.xpath('//meta[@property="og:title"]/@content').get() or ""
        description = w3lib.html.remove_tags(response.xpath('//meta[@property="og:description"]/@content').get()) or ""
       
        return {
            'id': self.total,
            'title': title,
            'description': description
        }

Is there a way we can restrict the crawler to crawl only the first 5 pages?

答案1

得分: 2

解决方案 1: 使用 process_request

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

def limit_requests(request, response):
    # 这里我们有页面编号。
    # page_number = request.url[-1]
    # if int(page_number) >= 6:
    #     return None

    # 这里我们使用一个计数器
    if not hasattr(limit_requests, "page_number"):
        limit_requests.page_number = 0
    limit_requests.page_number += 1

    if limit_requests.page_number >= 5:
        return None

    return request

class ExampleSpider(CrawlSpider):
    name = 'example_spider'

    start_urls = ['https://scrapingclub.com/exercise/list_basic/']
    page = 0
    rules = (
        # 获取一页上所有文章的列表并跟踪这些链接
        Rule(LinkExtractor(restrict_xpaths='//div[@class="card-body"]/h4/a'), callback="parse_item",
             follow=True),
        # 然后获取分页的下一页链接的 href 并跟踪它,重复这个循环
        Rule(LinkExtractor(restrict_xpaths='//li[@class="page-item"][last()]/a'), process_request=limit_requests)
    )
    total = 0

    def parse_item(self, response):
        title = response.xpath('//h3//text()').get(default='')
        price = response.xpath('//div[@class="card-body"]/h4//text()').get(default='')
        self.total = self.total + 1

        return {
            'id': self.total,
            'title': title,
            'price': price
        }

解决方案 2: 重写 _requests_to_follow 方法(虽然可能较慢)。

from scrapy.http import HtmlResponse
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ExampleSpider(CrawlSpider):
    name = 'example_spider'

    start_urls = ['https://scrapingclub.com/exercise/list_basic/']

    rules = (
        # 获取一页上所有文章的列表并跟踪这些链接
        Rule(LinkExtractor(restrict_xpaths='//div[@class="card-body&quot]/h4/a'), callback="parse_item",
             follow=True),
        # 然后获取分页的下一页链接的 href 并跟踪它,重复这个循环
        Rule(LinkExtractor(restrict_xpaths='//li[@class="page-item"][last()]/a'))
    )
    total = 0
    page = 0

    def _requests_to_follow(self, response):
        if not isinstance(response, HtmlResponse):
            return
        if self.page >= 5:  # 停止条件
            return
        seen = set()
        for rule_index, rule in enumerate(self._rules):
            links = [
                lnk
                for lnk in rule.link_extractor.extract_links(response)
                if lnk not in seen
            ]
            for link in rule.process_links(links):
                if rule_index == 1: # 假设只有一个“下一页”按钮
                    self.page += 1
                seen add(link)
                request = self._build_request(rule_index, link)
                yield rule.process_request(request, response)

    def parse_item(self, response):
        title = response.xpath('//h3//text()').get(default='')
        price = response.xpath('//div[@class="card-body&quot]/h4//text()').get(default='')
        self.total = self.total + 1

        return {
            'id': self.total,
            'title': title,
            'price': price
        }

这些解决方案都是比较自解释的,如果您需要添加其他内容,请在评论中提问。

英文:

Solution 1: use process_request.

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


def limit_requests(request, response):
    # here we have the page number.
    # page_number = request.url[-1]
    # if int(page_number) >= 6:
    #     return None

    # here we use a counter
    if not hasattr(limit_requests, "page_number"):
        limit_requests.page_number = 0
    limit_requests.page_number += 1

    if limit_requests.page_number >= 5:
        return None

    return request


class ExampleSpider(CrawlSpider):
    name = 'example_spider'

    start_urls = ['https://scrapingclub.com/exercise/list_basic/']
    page = 0
    rules = (
        # Get the list of all articles on the one page and follow these links
        Rule(LinkExtractor(restrict_xpaths='//div[@class="card-body"]/h4/a'), callback="parse_item",
             follow=True),
        # After that get pagination next link get href and follow it, repeat the cycle
        Rule(LinkExtractor(restrict_xpaths='//li[@class="page-item"][last()]/a'), process_request=limit_requests)
    )
    total = 0

    def parse_item(self, response):
        title = response.xpath('//h3//text()').get(default='')
        price = response.xpath('//div[@class="card-body"]/h4//text()').get(default='')
        self.total = self.total + 1

        return {
            'id': self.total,
            'title': title,
            'price': price
        }

Solution 2: overwrite _requests_to_follow method (should be slower though).

from scrapy.http import HtmlResponse
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class ExampleSpider(CrawlSpider):
    name = 'example_spider'

    start_urls = ['https://scrapingclub.com/exercise/list_basic/']

    rules = (
        # Get the list of all articles on the one page and follow these links
        Rule(LinkExtractor(restrict_xpaths='//div[@class="card-body"]/h4/a'), callback="parse_item",
             follow=True),
        # After that get pagination next link get href and follow it, repeat the cycle
        Rule(LinkExtractor(restrict_xpaths='//li[@class="page-item"][last()]/a'))
    )
    total = 0
    page = 0
    
    def _requests_to_follow(self, response):
        if not isinstance(response, HtmlResponse):
            return
        if self.page >= 5:  # stopping condition
            return
        seen = set()
        for rule_index, rule in enumerate(self._rules):
            links = [
                lnk
                for lnk in rule.link_extractor.extract_links(response)
                if lnk not in seen
            ]
            for link in rule.process_links(links):
                if rule_index == 1: # assuming there's only one "next button"
                    self.page += 1
                seen.add(link)
                request = self._build_request(rule_index, link)
                yield rule.process_request(request, response)

    def parse_item(self, response):
        title = response.xpath('//h3//text()').get(default='')
        price = response.xpath('//div[@class="card-body"]/h4//text()').get(default='')
        self.total = self.total + 1

        return {
            'id': self.total,
            'title': title,
            'price': price
        }

The solutions are pretty much self explanatory, if you want me to add something please ask in the comments.

huangapple
  • 本文由 发表于 2023年4月11日 16:29:05
  • 转载请务必保留本文链接:https://go.coder-hub.com/75983861.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定