Многостраничный парсинг сайтов

Question

import csv

from bs4 import BeautifulSoup as bs

# URL STEAMTP: https://steamcommunity.com/market/search?appid=730#p2_popular_desc
# style A "market_listing_row_link"

i = 1
while i < 5:
    url = ('https://steamcommunity.com/market/search?appid=730#p' + str(i) + '_popular_desc')
    r = requests.get(url)
    i += 1
    soup = bs(r.content, "html.parser")

    # print(soup.prettify())

    date = soup.find_all('a', class_='market_listing_row_link')
    for el in date:
        print(el.get("href"))

Пытаюсь пропарсить сайт Steam TM, и вытащить все ссылки. Но во время парсинга не выполняется переход на следующую страницу.

PS C:\Users\Smple\Documents\WebProject\trade> py main.py
https://steamcommunity.com/market/listings/730/Glove%20Case
https://steamcommunity.com/market/listings/730/Clutch%20Case
https://steamcommunity.com/market/listings/730/Operation%20Breakout%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Prisma%202%20Case
https://steamcommunity.com/market/listings/730/Gamma%202%20Case
https://steamcommunity.com/market/listings/730/Operation%20Phoenix%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Shattered%20Web%20Case
https://steamcommunity.com/market/listings/730/MP9%20%7C%20Orange%20Peel%20%28Field-Tested%29
https://steamcommunity.com/market/listings/730/Spectrum%20Case
https://steamcommunity.com/market/listings/730/Prisma%20Case
https://steamcommunity.com/market/listings/730/Glove%20Case
https://steamcommunity.com/market/listings/730/Clutch%20Case
https://steamcommunity.com/market/listings/730/Operation%20Breakout%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Prisma%202%20Case
https://steamcommunity.com/market/listings/730/Gamma%202%20Case
https://steamcommunity.com/market/listings/730/Operation%20Phoenix%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Shattered%20Web%20Case
https://steamcommunity.com/market/listings/730/MP9%20%7C%20Orange%20Peel%20%28Field-Tested%29
https://steamcommunity.com/market/listings/730/Spectrum%20Case
https://steamcommunity.com/market/listings/730/Prisma%20Case
https://steamcommunity.com/market/listings/730/Glove%20Case
https://steamcommunity.com/market/listings/730/Clutch%20Case
https://steamcommunity.com/market/listings/730/Operation%20Breakout%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Prisma%202%20Case
https://steamcommunity.com/market/listings/730/Gamma%202%20Case
https://steamcommunity.com/market/listings/730/Operation%20Phoenix%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Shattered%20Web%20Case
https://steamcommunity.com/market/listings/730/MP9%20%7C%20Orange%20Peel%20%28Field-Tested%29
https://steamcommunity.com/market/listings/730/Spectrum%20Case
https://steamcommunity.com/market/listings/730/Prisma%20Case
https://steamcommunity.com/market/listings/730/Glove%20Case
https://steamcommunity.com/market/listings/730/Clutch%20Case
https://steamcommunity.com/market/listings/730/Operation%20Breakout%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Prisma%202%20Case
https://steamcommunity.com/market/listings/730/Gamma%202%20Case
https://steamcommunity.com/market/listings/730/Operation%20Phoenix%20Weapon%20Case
https://steamcommunity.com/market/listings/730/Shattered%20Web%20Case
https://steamcommunity.com/market/listings/730/MP9%20%7C%20Orange%20Peel%20%28Field-Tested%29
https://steamcommunity.com/market/listings/730/Spectrum%20Case
https://steamcommunity.com/market/listings/730/Prisma%20Case
PS C:\Users\Smple\Documents\WebProject\trade>

Как видно парсер парсит 1 страницу 3 раза. А нужно пропарсить 1 страницу 2 страницу и 3 страницу.

Answer 1

На самом деле там используется отдельный ajax-запрос на сервер (это нужно смотреть в панели инспектора браузера на вкладке сети):

При загрузке страницы с номером. В этом случае загрузится первая страница, а после будет сделан ajax-запрос на сервер с указанием данных
При переходе на страницу через панель навигации. В этом случае не будет новой загрузки страницы, а только ajax-запрос и замена якоря в url (я про #p4_popular_desc)

Его url выглядит вот так:

https://steamcommunity.com/market/search/render/?query=&start=10&count=10&search_descriptions=0&sort_column=popular&sort_dir=desc&appid=730

А в ответ придет такое:

{'success': True, 'start': 10, 'pagesize': 10, 'total_count': 14987, 'tip': "Set norender=1 if you don't want HTML", 'results_html': '<кусок_html>'}

Накидал пример. По хорошему, нужно анализировать что приходит в ответе в поле total_count и на его основе, плюс с значением в переменной start решать нужно ли еще в цикле работать:

from bs4 import BeautifulSoup
import requests

start = 0

i = 0
while i < 5:
    i += 1

    url = f'https://steamcommunity.com/market/search/render/?query=&start={start}&count=10&search_descriptions=0&sort_column=popular&sort_dir=desc&appid=730'
    rs = requests.get(url)

    html = rs.json()['results_html']
    root = BeautifulSoup(html, "html.parser")

    for a in root.select('.market_listing_row_link'):
        print(a["href"])

    print('\n' + '-' * 100 + '\n')

    start += 10

Результат первых двух итераций:

# https://steamcommunity.com/market/listings/730/Glove%20Case
# https://steamcommunity.com/market/listings/730/Clutch%20Case
# https://steamcommunity.com/market/listings/730/Operation%20Breakout%20Weapon%20Case
# https://steamcommunity.com/market/listings/730/Prisma%202%20Case
# https://steamcommunity.com/market/listings/730/Gamma%202%20Case
# https://steamcommunity.com/market/listings/730/Operation%20Phoenix%20Weapon%20Case
# https://steamcommunity.com/market/listings/730/Shattered%20Web%20Case
# https://steamcommunity.com/market/listings/730/MP9%20%7C%20Orange%20Peel%20%28Field-Tested%29
# https://steamcommunity.com/market/listings/730/Spectrum%20Case
# https://steamcommunity.com/market/listings/730/Prisma%20Case
#
# ----------------------------------------------------------------------------------------------------
#
# https://steamcommunity.com/market/listings/730/Chroma%202%20Case
# https://steamcommunity.com/market/listings/730/Spectrum%202%20Case
# https://steamcommunity.com/market/listings/730/Five-SeveN%20%7C%20Coolant%20%28Field-Tested%29
# https://steamcommunity.com/market/listings/730/Gamma%20Case
# https://steamcommunity.com/market/listings/730/M4A1-S%20%7C%20Decimator%20%28Field-Tested%29
# https://steamcommunity.com/market/listings/730/Danger%20Zone%20Case
# https://steamcommunity.com/market/listings/730/Desert%20Eagle%20%7C%20Mecha%20Industries%20%28Minimal%20Wear%29
# https://steamcommunity.com/market/listings/730/SSG%2008%20%7C%20Dragonfire%20%28Field-Tested%29
# https://steamcommunity.com/market/listings/730/Berlin%202019%20Minor%20Challengers%20Autograph%20Capsule
# https://steamcommunity.com/market/listings/730/CS20%20Case
#
# ...
#

БЛОГ НА HUSL

Многостраничный парсинг сайтов

Ответы (1 шт):