A node js package for scraping the chrome web store.
This project requires selenium, a Web Browser Automation tool. The latest version of the Selenium Standalone Server can be downloaded from seleniumhq.
Selenium Server must also be installed as
selenium on the system PATH. For linux, a
selenium bash script is included that can be paired with the
selenium.jar for ease of use.
The selenium-webdriver npm package has some details on what is required.
The Selenium server must be on the system path as '
selenium' the easiest way to set it up to work with the chrome web store scraper is to make the
selenium bash script (that is included this project) an executable with
chmod +x selenium and then copy that file, along with the selenium server
.jar file to
/bin/ or somewhere similar.
When copying the selenium server
.jar make sure it is renamed from
selenium-server-standalone-3.14.0.jar or whatever it is currently called, to just
As well as selenium, you're going to need the latest chromedriver installed.
A chrome browser is also required. you can get the latest chrome-broswer-stable from chromium.
How To Use
You can use this to scrape search results for chrome extensions, or to scrape store information for a specific extension.
To include the scraper in your project:
const ChromeWebScraper = ;const scraper = ;
The most basic search just requires you to provide a search term.
there are additional options than can be used to perform a more directed search
You can provide as a category which the scraper will then use when building a search request, only one category can be provided.
Categories can be provided in an options JSON object as demonstrated below:
const options = searchCategory : 'newsAndWeather'scraper;
Search features can be provided as a means of specifying select features that a chrome extension must have.
Passed as an array of strings in an options JSON object, the features can be any combination of the following:
features can be provided in an options JSON object as demonstrated below:
const options =searchFeatures : 'free' 'offline''byGoogle'scraper;
Features and Categories
Searching can be performed with categories and features together in the same options JSON object.
const options =searchCategory : 'newsAndWeather'searchFeatures : 'free' 'byGoogle'scraper;
As well as the categories and feature filters, additional options in the form of
scrollAttempts can also be used.
scrollAttempts option is used to specify how many attempts are made to retrieve additional search results, by scrolling down the page loaded by selenium. the larger the number, the more the page will be scrolled down. It is worth noting that each scroll attempt is paired with a 50ms wait, so a number excessively large will result in longer processing times.
locale option can be set by passing a locale string as an option. The locale string is used as the
hl url option by the chrome extension store, to scrape search results in french, danish, or italian, locale strings
'it' could be used.
Example Search with locale and scrollAttempts
In order to scrape the store page for a specific chrome extension, this scraper requires a direct url to that page. These urls are to be passed as a parameter to the