Javascript Compiling Tokenizer

Give your Javascript the token of love

Hello, welcome to JavaScript tokenizer

This project has the sole purpose of tokenizing JavaScript by traversing a given file (stringified) and generating an abstract syntax tree from the resulting tokenization process

JCT does not worry about loading the files or what to do with the files after pushing them into an abstract syntax tree it simply concerns itself with the process of generating an AST

Other projects utilise this projects and take the resulting abstract syntax tree and transpire the JavaScript accordingly

For example require to ecma takes the abstract syntax tree generated from this code and uses it to transpile code that contains the old require import system into the new es6 import system

usage

Install it

    npm i javascript-compiling-tokenizer

Import it

    import { LexicalAnalyzer, Generator } from 'javascript-compiling-tokenizer';

Initialize it

    const tokenizer = new LexicalAnalyzer(options)

    //options only have one property.... 'verbose' [boolean]

Tokenize it it

    const syntaxTree = tokenizer().start(fileAsString);

Turn it back into javascript

    const js = new Generator().start(syntaxTree);

Test it

    npm test

options

{
    verbose: boolean //will log to console
    thirdPartyParsingTests: Array<(char: string, current: number, input: string) => IThirdPartyParsingResult> = [];
}

Injecting your own third party lexical checks

Its as simple as providing an array of functions. The functions are given the following arguments:

char:string, current:number, input:string

char is the string at the current position, current is the index of the current position in the input string and input is a stringified version of the whole file.

The function must return an object (IThirdPartyParsingResult) containing:

{
    payload: {type: 'coolnew type', value: 'the value'} //the token
    current: number //new cursor position after going through this function
}

Third party lexical checks are always performed first.

Transforming tokens into Javascript

   new Generator().start(tokens)

This will return a string representation of the tokens provided. all you have to do is pipe it into a file.

Dev notes

The tokenizer will recurse in the following conditions:

• if it finds an opening parenthesis (
• if it finds an opening code block {
• if it finds an opening array [
• if it finds a declaration const, let, var

Examples

There are more examples of this being used on a react file, normal js file and a file containing the 'old' 'defines' method to import things in ./tests/beforeandafter

Gotchas

• This tokenizer wont parse files correctly if variables are not declared properly. If you declare variables without a declaration statement such as const, var, let. Then this tokenizer is not for you.
• Parser doesn't support comma separated declarations. let var1,var2,var3,var4; will not parse correctly.

Some regex

   import * as _ from 'underscore';
   import * as colors from 'colors';
 
   const CARRIAGE_RETURN = /\n/;
   const EOL = /\r/;
   const WHITESPACE = /\s/;
   const NUMBERS = /[0-9]/;
   const DECLARABLE_CHARACTERS = /[A-Za-z_.$]/i;

The AST

   {
    "tokens": [{
        "type": "name",
        "value": "import"
    }, {
        "type": "operator",
        "value": "*"
    }, {
        "type": "name",
        "value": "as"
    }, {
        "type": "name",
        "value": "_"
    }, {
        "type": "name",
        "value": "from"
    }, {
        "type": "string",
        "value": "underscore"
    }, {
        "type": "statementSeperator",
        "value": ";"
    }, {
        "type": "carriagereturn",
        "value": 1
    }, {
        "type": "name",
        "value": "import"
    }, {
        "type": "operator",
        "value": "*"
    }, {
        "type": "name",
        "value": "as"
    }, {
        "type": "name",
        "value": "colors"
    }, {
        "type": "name",
        "value": "from"
    }, {
        "type": "string",
        "value": "colors"
    }, {
        "type": "statementSeperator",
        "value": ";"
    }, {
        "type": "carriagereturn",
        "value": 2
    }, {
        "type": "carriagereturn",
        "value": 3
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "CARRIAGE_RETURN"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "assignee",
            "value": "/\\n/"
        }]
    }, {
        "type": "carriagereturn",
        "value": 4
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "EOL"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "assignee",
            "value": "/\\r/"
        }]
    }, {
        "type": "carriagereturn",
        "value": 5
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "WHITESPACE"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "assignee",
            "value": "/\\s/"
        }]
    }, {
        "type": "carriagereturn",
        "value": 6
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "NUMBERS"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "assignee",
            "value": "/[0-9]/"
        }]
    }, {
        "type": "carriagereturn",
        "value": 7
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "DECLARABLE_CHARACTERS"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "assignee",
            "value": "/[A-Za-z_.$]/i"
        }]
    }],
    "current": 211
}

The generated code

(its the same...)

   import * as _ from 'underscore';
   import * as colors from 'colors';
 
   const CARRIAGE_RETURN = /\n/;
   const EOL = /\r/;
   const WHITESPACE = /\s/;
   const NUMBERS = /[0-9]/;
   const DECLARABLE_CHARACTERS = /[A-Za-z_.$]/i;

Trixie code

   if(true === false && !!(false >= true) || false != true)
   {
      let result = (((5*5)%5)/5);
      console.log('test', () => { return /** random inline comment */ !!true})
      //inline multichar ternary/comparator/operator test
   }

AST

   {
    "tokens": [{
        "type": "name",
        "value": "if"
    }, {
        "type": "params",
        "value": [{
            "type": "name",
            "value": "true "
        }, {
            "type": "operator",
            "value": "==="
        }, {
            "type": "assignee",
            "value": "false"
        }, {
            "type": "operator",
            "value": "&&"
        }, {
            "type": "operator",
            "value": "!!("
        }, {
            "type": "name",
            "value": "false "
        }, {
            "type": "operator",
            "value": ">="
        }, {
            "type": "name",
            "value": "true"
        }]
    }, {
        "type": "operator",
        "value": "||"
    }, {
        "type": "name",
        "value": "false "
    }, {
        "type": "operator",
        "value": "!="
    }, {
        "type": "name",
        "value": "true"
    }, {
        "type": "operator",
        "value": ")"
    }, {
        "type": "carriagereturn",
        "value": "\n"
    }, {
        "type": "codeblock",
        "value": [{
            "type": "carriagereturn",
            "value": "\n"
        }, {
            "type": "let",
            "value": [{
                "type": "name",
                "value": "result "
            }, {
                "type": "assigner",
                "value": "="
            }, {
                "type": "params",
                "value": [{
                    "type": "params",
                    "value": [{
                        "type": "params",
                        "value": [{
                            "type": "assignee",
                            "value": "5*5"
                        }]
                    }, {
                        "type": "operator",
                        "value": "%"
                    }, {
                        "type": "name",
                        "value": "5"
                    }]
                }, {
                    "type": "operator",
                    "value": "/"
                }, {
                    "type": "name",
                    "value": "5"
                }]
            }, {
                "type": "statementseperator",
                "value": ";"
            }]
        }, {
            "type": "carriagereturn",
            "value": "\n"
        }, {
            "type": "name",
            "value": "console.log"
        }, {
            "type": "params",
            "value": [{
                "type": "string",
                "value": "'test'"
            }, {
                "type": "seperator",
                "value": ","
            }, {
                "type": "params",
                "value": []
            }, {
                "type": "assigner",
                "value": "="
            }, {
                "type": "assignee",
                "value": ">"
            }, {
                "type": "codeblock",
                "value": [{
                    "type": "name",
                    "value": "return "
                }, {
                    "type": "multilinecomment",
                    "value": " ** random inline comment "
                }, {
                    "type": "operator",
                    "value": "!!"
                }, {
                    "type": "name",
                    "value": "true"
                }]
            }]
        }, {
            "type": "carriagereturn",
            "value": "\n"
        }, {
            "type": "carriagereturn",
            "value": "\n"
        }, {
            "type": "inlinecomment",
            "value": "//inline multichar ternary/comparator/operator test"
        }, {
            "type": "carriagereturn",
            "value": "\n"
        }]
    }],
    "current": 225
}

The output

if(true === false && !!(false >= true) || false != true)
{
    let  result = (((5*5)%5)/5);
    console.log('test', () => { return /** random inline comment */ !!true})
    
//inline multichar ternary/comparator/operator test
}

Some string literals

   const string = `something ${1 + 2 + 3}`;
   const string2 = `something ${(true) ? 'x' : 'y'}`;
   const string3 = `something
   another ${'x'}
   new line ${1 + 2 + 3}
   test`;

the AST

   {
   {
    "tokens": [{
        "type": "carriagereturn",
        "value": 1
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "string"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "stringLiteral",
            "value": "something ${1 + 2 + 3}"
        }]
    }, {
        "type": "carriagereturn",
        "value": 2
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "string"
        }, {
            "type": "number",
            "value": "2"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "stringLiteral",
            "value": "something ${(true) ? 'x' : 'y'}"
        }]
    }, {
        "type": "carriagereturn",
        "value": 3
    }, {
        "type": "const",
        "value": [{
            "type": "name",
            "value": "string"
        }, {
            "type": "number",
            "value": "3"
        }, {
            "type": "assigner",
            "value": "="
        }, {
            "type": "stringLiteral",
            "value": "something\nanother ${'x'}\nnew line ${1 + 2 + 3}\ntest"
        }]
    }],
    "current": 163
}