Tokenizing Python with Racket and libpython
I wrote a little tokenizer for Python – well, rather, some Racket code that calls out to the native Python tokenizer. This was a fun excercise, becase it could allow for micro-interpreters to be written for Python or Python-like languages like bazel in Racket.
Check it out if you’re interested https://github.com/abhillman/py-tokenizer-rkt.