Scrapping data not coming from exact url


I’m trying to scrap some monster infobox table from rswiki.

Some specific monster have multiple levels, for example:

You can switch through the different levels by clicking on boxes on top of the infobox: "Level 7","Level 10"…

Once you click on the level box it changes the url to match the level.

So when i request the url, it’s bringing data from the first level only, in case:, and i can’t get to scrap other levels.

import requests
from bs4 import BeautifulSoup

url = ''
response = requests.get(url, headers = {'User-Agent':'Mozilla/5.0'})
soup = BeautifulSoup(response.content, 'html.parser')
soup_minfobox = soup.find_all('table', class_ ="infobox infobox-switch no-parenthesis-style infobox-monster")


Output: Level 7Level 10Level 11Level 20DwarfReleased6 April 2001 (Update)MembersNoCombat level7Size1x1 …

Excuse me the makeshift code, but in the output you can see that it is the data from the lv 7 in the end, although the url is for the lv 20.


If you manually trigger the events (from the browser’s console), you’ll see that the infobox changes:


So you can use the above selectors and consult the answers provided in the following topic on how to invoke an event using BeautifulSoup:

invoking onclick event with beautifulsoup python

Answered By – Nick Papadiochos

Answer Checked By – Mary Flores (AngularFixing Volunteer)

Leave a Reply

Your email address will not be published.