Web Scraping Gaming Data from Steam Platform Using Python

Check the early-stage code below.

After the above steps, we will make a function to search the last page of Steam game data we are hunting, namely get_pagnition(). Firstly we create a param variable in this function. The param variable is similar to the head variable in terms of functionality. When we call the Requests library, call the function as an additional parameter. There are two values in the param function. The first term uses game keywords as value, and the second is a page with one as the value. We kept the one as the value because we wanted to retrieve only one page. Depending on requirements, the page will change the available page count in the following function.

Then, we will create a req variable that contains the Requests library and get function. There are extra headers and params parameters. Later, we create a soup variable to input the req.text variable, locate the BeautifulSoup library, and use the HTML parser. Then, we search for the page tag location.

If we check the above HTML structure, we should find the search_pagination_right class first, then find_all tag to get each number in the search_pagination_right class. After that, we save this function in another variable, page_item, to call it. Since the variable page_item is near the finishing find_all, this variable's data type is the list. We use the error handling to get the last page_item list number when the number in the fifth list is the largest. We keep doing the same process until we get the last number. We will take the one as the order value if there is only one page. Consider that we can return one extra page than total_page. Then we must ass our goals to the first number because computer calculations begin from zero. We should add 1 to get the number according to our expectations.

Check the code below for the get_pagination function we wrote.

In the next step, we make a web scraping function of each data, namely scrap(). It has the role of creating a serial number for every data available to us. Additionally, we create a data variable using a list type of data to serve as a data container. We use a loop to access each page with two ranges to the function get_pagnition we initially created. Firstly we make a counter variable with 0 to make the scrap function. Then we make a param variable with a value similar to the param variable available inside the get_pagination() function. Then, we make soup and req variables similar to those in the get_pagination(). Since we are using a loop, the page item has a rising value.

Now, it is time to collect all the required data. Firstly we should see the target tag that we will use.

Considering the above HTML structure, every fame data is available in the div tag with the search_resultsRows id. Every game exists in the tag, so we discover the div tag with search_resultsRows id. Then, we find every tag to get all the game data and locate it in the content variable. After getting all the game data, we parse it according to our needs as we use the loop since data is available as a list inside the content variable. We will collect titles, links, release dates, and price data. After getting the data, we feed it to the dictionary variable and put the dictionary in the data variable that we initially created. The function will simplify extracting the gaming data in Excel, CSV, and JSON files. Make sure to use the syntax i+=1 as a continuously growing counter. Finally, we will print all the scraped data per the following code.

While extracting title data, we have used the strip and replace syntax, which deletes the space available in front of the text and removes newlines in all data. Besides, while collecting pricing data, we have used the try-except function using the condition that we can take the data from the discount tag if there is a discount on the game. We follow the same process while scraping date-release data. We only take a value if price information is available while scraping price data.

We've completed the process to scrape Steam game data and can now extract it. To extract it, you need to write and execute the code without using a loop to avoid repetition of extraction. Initially, we will extract the data in a JSON file. Then we retrieve it in the CSV and Excel files. For this, we use the pandas library. You can expect a change in the file name because of the game variable. Give a false value to the index. Remember to install the openpyxl package to retrieve data using pandas.

Conclusion

This way, we have created a Python code to scrape Steam game data. If you want a customized solution for web scraping gaming data from Steam, contact Actowiz Solutions.