333 packages found

M
Q
P

Description

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

Keywords

Publisher

published 1.2.1a year ago
M
Q
P

Description

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

Keywords

Publisher

published 2.0.06 months ago
M
Q
P

Description

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Keywords

Publisher

published 2.0.04 months ago
M
Q
P

Description

Simple WAF to integrate with Node.js web systems

Keywords

Publisher

published 1.0.35 months ago
M
Q
P

Description

极简网络蜘蛛爬虫,适用任何网站,只需设置一条规则,就可以把你想要网站上的内容整理出来,非常方便,简单!

Keywords

Publisher

published 5.0.109 months ago
M
Q
P

Description

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

Keywords

Publisher

published 3.0.12 months ago
M
Q
P

Description

crawlx is a Lightweight web crawler with powerful plugins!

Keywords

Publisher

published 1.5.12 months ago
M
Q
P

Description

xvideos.com api implementation and update.

Keywords

Publisher

published 1.1.195 months ago
M
Q
P

Description

And ExpressJS middleware for detecting search engine crawlers/spiders and handling crawler-specific logic such as additional logging for requests from spiders

Keywords

Publisher

published 1.1.610 months ago
M
Q
P

Description

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

Keywords

Publisher

published 1.1.810 months ago
M
Q
P

Description

Crawler (spider) of site web pages by domain name

Keywords

Publisher

published 1.0.78 months ago
M
Q
P

Description

DEX8 SDK is a library which helps developers to create DEX8 crawler, scraper tasks or automated serverless scripts.

Keywords

Publisher

published 2.0.94 days ago
M
Q
P

Description

Crawl data or download files from website with custom rules

Keywords

Publisher

published 1.0.24 months ago
M
Q
P

Description

node.js web crawler

Keywords

Publisher

published 2.0.17 months ago
M
Q
P

Description

A really small crawler.

Keywords

Publisher

published 3.0.08 months ago
M
Q
P

Description

A node crawler that return all links/href from website

Keywords

Publisher

published 1.0.23 months ago
M
Q
P

Description

xvideos.com api implementation.

Keywords

Publisher

published 1.2.09 months ago
M
Q
P

Description

A aliexpress spider

Keywords

Publisher

published 0.2.39 months ago
M
Q
P

Description

make up a web spider with a few regular module。

Keywords

Publisher

published 0.3.114 days ago
M
Q
P

Description

爬虫工具,根据传入的配置和规则爬取数据

Keywords

Publisher

published 1.1.34 months ago