genscrape

    1.5.2 • Public • Published

    npm Build Status Coverage Status

    genscrape

    A JavaScript library that aids in scraping person data off of genealogy websites. It is designed to be used in a browser extension. Data is output in GEDCOM X format.

    Install

    npm install --save genscrape
    

    Or use the CDN

    <script src="https://unpkg.com/genscrape@latest/dist/genscrape.min.js"></script>

    We recommend pinning CDN links to a specific version.

    Usage

    Scraping is sometimes asynchronous so we chose to implement the EventEmitter interface using EventEmmitter2.

    genscrape().on('data', function(data){
      // Do something with the data
    });

    That's it. Genscrape automatically detects what page you're on, looks up the correct scraper, then does its magic.

    Events

    data

    The data event is fired when genscrape successfully scrapes data from the page. Read more about the data model below.

    noMatch

    The noMatch event is fired when genscrape is called on a page with a URL that it does not recognize. In other words, it does not have a parser registered for the URL.

    noData

    The noData event is fired when genscrape is run on a page that a parser is registered for but the parser was unable to find any data.

    In cases of one-page apps such as the FamilySearch Family Tree and the findmypast trees, you may see multiple noData and data events as the user browses through the tree.

    error

    The error event is fired when something unexpected occurs while a scraper is processing, such as a failed AJAX call. There is no standardized format for the errors yet.

    Data Model

    Data is returned in the GEDCOM X JSON format.

    When possible, we mark one person as principal. This isn't possible when viewing marriage records on sites that don't make a distinction between persons in a record. The only currently supported site where this occurs is Open Archives.

    Every GEDCOM X document has a SourceDescription, pointed to by the about property of the document, that provides a citation for all data in the document. That SourceDescription points to an Agent, via the repository property, which describes the website the data came from.

    {
      "id": "agent",
      "names": [
        {
          "lang": "en",
          "value": "Find A Grave"
        }
      ],
      "homepage": {
        "resource": "https://www.findagrave.com"
      }
    }

    We populate person's IDs with IDs from the source repository.

    Person IDs aren't unique between websites. Thus we add a genscrape Identifier which allow us to better compare two arbitrary records to determine if they come from the same record on the same website.

    {
      "id": "65630115",
      "identifiers": {
        "genscrape": [
          "genscrape://findagrave/65630115"
        ]
      }
    }

    Supported Sites

    Keywords

    Install

    npm i genscrape

    DownloadsWeekly Downloads

    4

    Version

    1.5.2

    License

    MIT

    Unpacked Size

    818 kB

    Total Files

    26

    Last publish

    Collaborators

    • avatar