npm

307 packages found

M
Q
P

Description

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

Keywords

Publisher

published 1.2.17 months ago
M
Q
P

Description

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

Keywords

Publisher

published 2.0.02 months ago
M
Q
P

Description

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

Keywords

Publisher

published 1.1.85 months ago
M
Q
P

Description

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Keywords

Publisher

published 1.7.22 months ago
M
Q
P

Description

极简网络蜘蛛爬虫,适用任何网站,只需设置一条规则,就可以把你想要网站上的内容整理出来,非常方便,简单!

Keywords

Publisher

published 5.0.104 months ago
M
Q
P

Description

Simple WAF to integrate with Node.js web systems

Keywords

Publisher

published 1.0.311 days ago
M
Q
P

Description

And ExpressJS middleware for detecting search engine crawlers/spiders and handling crawler-specific logic such as additional logging for requests from spiders

Keywords

Publisher

published 1.1.65 months ago
M
Q
P

Description

Crawler (spider) of site web pages by domain name

Keywords

Publisher

published 1.0.74 months ago
M
Q
P

Description

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

Keywords

Publisher

published 3.0.05 months ago
M
Q
P

Description

xvideos.com api implementation and update.

Keywords

Publisher

published 1.1.19a month ago
M
Q
P

Description

xvideos.com api implementation.

Keywords

Publisher

published 1.2.05 months ago
M
Q
P

Description

爬虫工具,根据传入的配置和规则爬取数据

Keywords

Publisher

published 1.0.8119 days ago
M
Q
P

Description

Crawl data or download files from website with custom rules

Keywords

Publisher

published 1.0.12 months ago
M
Q
P

Description

Utils for web resources. Get a web page and save to disk (with minimal dependencies)

Keywords

Publisher

published 1.0.02 months ago
M
Q
P

Description

node.js web crawler

Keywords

Publisher

published 2.0.13 months ago
M
Q
P

Description

A aliexpress spider

Keywords

Publisher

published 0.2.35 months ago
M
Q
P

Description

A really small crawler.

Keywords

Publisher

published 3.0.03 months ago
M
Q
P

Description

a node spider framework

Keywords

Publisher

published 1.0.11a year ago
M
Q
P

Description

web crawler, parser and scraper with storage capabilities

Keywords

Publisher

published 0.3.8a year ago
M
Q
P

Description

Simple, flexible, delightful web crawler/spider package

Keywords

Publisher

published 0.11.44 months ago