JavaScript has become ubiquitous on the web, posing a challenge for web scraping. While most data is easily accessible in the HTML of a page, there are instances where the data is only available after the JavaScript is executed and rendered. This complicates the scraping process.
In previous articles, we have covered web scraping using user-friendly Python libraries. However, these methods need to be improved when dealing with JavaScript rendering. We need to employ more advanced tools and techniques to tackle websites that hide their data behind JavaScript rendering.
Normally, we would suggest some go-to libraries to do web scraping:
BeautifulSoup
Requests
While the previously mentioned tools can handle various tasks, they cannot render JavaScript. Fortunately, there is a dedicated category of tools designed specifically for this purpose called Browser Automation tools.
Browser Automation tools are built to simulate and automate the web browsing experience, allowing tasks to be executed at intervals or speeds surpassing human capabilities. While these tools are commonly used for website testing by owners, they also offer the functionality required to render JavaScript and scrape the underlying data.
These tools provide:
A comprehensive solution.
Combining JavaScript rendering capabilities with web scraping functionality.
Making them ideal for extracting data from JavaScript-intensive websites.
By leveraging Browser Automation tools, you can effectively overcome the challenge of scraping websites that rely heavily on JavaScript.
Among the popular tools are:
For the purpose of this example, let's delve into using Selenium. Additionally, we will utilize the reliable BeautifulSoup library to parse the response and extract the desired data.
To fully automate a web browser, some additional setup is necessary beyond the basic installation of libraries. We will need to install the following components:
Chrome Browser (or any other web browser of your choice, but we will use Chrome for this example).
ChromeDriver: This is the web driver specific to Chrome that allows interaction with the browser programmatically.
To follow along, you can install Chrome on your system. As for ChromeDriver installation, we can use a convenient Python library that handles the installation for us.
With that in mind, let's proceed with installing the required libraries.
Once they get installed, we could start importing:
To simplify the installation process, we can use the chromedriver_autoinstaller library, which automatically installs ChromeDriver and adds it to the system's PATH if it's not already present. This saves us some effort and can be achieved with a single line of code:
By executing this code, the library will handle the installation of ChromeDriver seamlessly.
Here's a summary of the steps we've taken to set up our environment. Assuming you have already installed Python, you can use pip to install the necessary libraries. After installing selenium, bs4, and chromedriver-autoinstaller, your Python file should look something like this:
Now that we have our environment set up, we can start making web page requests. To accomplish this, we need to configure the WebDriver object that Selenium will use. Here's an example of how you can set it up:
Now we can instruct the webdriver to retrieve a web page. For this example, we'll scrape Rotten Tomatoes Certified Fresh Movies.
Although the data we want (movie titles, ratings, etc.) can be obtained without rendering JavaScript, it's much easier to parse when it's rendered.
This page heavily relies on JavaScript, as shown in the JavaScript-enabled site:
And using JavaScript disabled:
We can ask for this web page using driver object “get” method
And we could get an html output with a page_source attribute:
Just to do recap, let’s go through the code:
Now, Selenium can handle parsing the data, but in most cases, we'll rely on BeautifulSoup for parsing the HTML. Let's create a BeautifulSoup object from the page source:
Now, we need to have something which will look like this:
If we were to print the soup object we've created, we would see the entire web page, excluding some of the fancy formatting. Fortunately, in this case, we don't have to wait for JavaScript execution.
However, in some scenarios, we may need to wait for JavaScript execution. This can be achieved either through Implicit Waiting or Explicit Waiting.
Since we don't need to worry about that here, let's focus on finding the information we're interested in:
It appears that all the movies we're interested in are contained within div elements with the class "mb-movie". Each of these divs contains information about an individual movie.
To extract the relevant information, we can use BeautifulSoup's find_all() method with the appropriate parameters:
We can have each of them and find a title, release date, and score and easily using BeautifulSoup:
In summary, we have accomplished several tasks in a short period of time:
Installed Chrome and ChromeDriver.
Used a Python library to install ChromeDriver automatically.
Fetched a web page that heavily relies on JavaScript using Selenium.
Parsed and extracted data from the web page using BeautifulSoup.
Here's a final overview of our progress, with the data printed in the terminal:
For more information, contact Actowiz Solutions now! Call us for all your mobile app scaping and data collection service requirements.
✨ "1000+ Projects Delivered Globally"
⭐ "Rated 4.9/5 on Google & G2"
🔒 "Your data is secure with us. NDA available."
💬 "Average Response Time: Under 12 hours"
Look Back Analyze historical data to discover patterns, anomalies, and shifts in customer behavior.
Find Insights Use AI to connect data points and uncover market changes. Meanwhile.
Move Forward Predict demand, price shifts, and future opportunities across geographies.
Industry:
Coffee / Beverage / D2C
Result
2x Faster
Smarter product targeting
Operations Manager, Beanly Coffee
✓ Competitive insights from multiple platforms
Industry:
Real Estate
Result
2x Faster
Real-time RERA insights for 20+ states
Data Analyst, Aditya Birla Group
✓ Boosted data acquisition speed by 3×
Industry:
Organic Grocery / FMCG
Result
Improved
competitive benchmarking
Product Manager, 24Mantra Organic
✓ Real-time SKU-level tracking
Industry:
Quick Commerce
Result
2x Faster
Inventory Decisions
Aarav Shah, Senior Data Analyst, Mensa Brands
✓ 28% product availability accuracy
✓ Reduced OOS by 34% in 3 weeks
Industry:
Quick Commerce
Result
3x Faster
improvement in operational efficiency
Business Development Lead,Organic Tattva
✓ Weekly competitor pricing feeds
Industry:
Beverage / D2C
Result
Faster
Trend Detection
Marketing Director, Sleepyowl Coffee
Boosted marketing responsiveness
Industry:
Quick Commerce
Result
Enhanced
stock tracking across SKUs
Growth Analyst, TheBakersDozen.in
✓ Improved rank visibility of top products
Real results from real businesses using Actowiz Solutions
In Stock
₹524
Price Drop + 12 min
in 6 hrs across Lel.6
Price
Drop −12 thr
Improved inventory
visibility & planning
Actowiz's real-time scraping dashboard helps you monitor stock levels, delivery times, and price drops across Blinkit, Amazon: Zepto & more.
✔ Scraped Data: Price Insights Top-selling SKUs
"Actowiz's helped us reduce out of stock incidents by 23% within 6 weeks"
✔ Scraped Data, SKU availability, delivery time
With hourly price monitoring, we aligned promotions with competitors, drove 17%
✔ Scraped Data, SKU availability, delivery time
"Actowiz's helped us reduce out of stock incidents by 23% within 6 weeks"
✔ Scraped Data, SKU availability, delivery time
Actionable Blogs, Real Case Studies, and Visual Data Stories -All in One Place
Unlock OTA growth with Scraping Booking.com Data for Competitive Pricing Analysis. Gain real-time insights, optimize pricing, and stay ahead of competitors.
Actowiz Solutions offers real-time Hermès Birkin bag availability & resale price tracking from top luxury platforms across USA, UAE, UK, and global markets.
reveals how brands Extract Festive Sale Data from Amazon, Flipkart & Reliance with 90% flash-sale alerts and 50+ brands analyzed.
Unlock OTA growth with Scraping Booking.com Data for Competitive Pricing Analysis. Gain real-time insights, optimize pricing, and stay ahead of competitors.
Unlock Sephora’s stock insights with Sephora Inventory & Stock Data Scraping API, tracking product availability across regions with 90–98% accuracy.
Discover weekly fee variations with Web Scraping weekly Delivery Fees Data From GrabFood, revealing PH, SG, and MY delivery costs shifting 10–25%.
Actowiz Solutions offers real-time Hermès Birkin bag availability & resale price tracking from top luxury platforms across USA, UAE, UK, and global markets.
how resorts Scrape Booking.com Seasonal Pricing Trends for Resorts to optimize peak season campaigns, maximize bookings, and drive revenue.
Real-Time Price Monitoring for Luxury Brands, highlighting Louis Vuitton, Gucci, and Prada across global markets with key pricing insights.
reveals how brands Extract Festive Sale Data from Amazon, Flipkart & Reliance with 90% flash-sale alerts and 50+ brands analyzed.
Explore Historical Navratri Sales Data from 2020–2025 to track discounts, flash sales, and consumer trends across Amazon, Flipkart, and Myntra.
Explore Myntra vs Ajio Navratri discount scraping insights for 2025—compare festive fashion offers, flash sales, and 2x shopper growth trends.
