AST Black Magic: Run synchronous Python code on asynchronous Pyodide

The ast module allows you to manipulate the AST (Abstract Syntax Tree) and change the behavior of the code at runtime.

I would like to introduce it with an interesting use case with Pyodide, a Python distribution for the browser. Pyodide has some differences from CPython due to its nature, so some code for CPython does not work on Pyodide. For example, you need to use await asyncio.sleep() instead of time.sleep() on Pyodide.

However, you may want to run the same code both on CPython and Pyodide for interoperability. In such a case, you can modify the AST to make it work on Pyodide without modifying the original Python code.

This talk should provide insights about how to use the ast module to solve some problems with actual examples.

I will introduce the ast module and how powerful the AST manipulation is by sharing my actual experience on Pyodide.

Pyodide is a Python distribution that runs in the browser, and it has some limitations as follows:

• time.sleep() uses a busy-wait and sometimes doesn't work as expected for example with yield
• asyncio.run() raises an error

So for better interoperability, the following code changes are needed:

• time.sleep() -> await asyncio.sleep()
• asyncio.run(fn()) -> await fn()

And if a function fn contains such method calls, the function definition must be changed to async def fn() because they are transformed to be awaited as above, so fn() must be changed to await fn() then.

• def fn() -> async def fn()
• fn() -> await fn()
• mod.fn() -> await mod.fn()

I actually implemented these code changes as AST transformations in my project, Stlite, which has a kind of a script runner that can run CPython-targeted code on Pyodide.

Its AST transformation code is available at https://github.com/whitphx/stlite/blob/main/packages/kernel/py/stlite-lib/stlite_lib/codemod.py. It's very complicated so I will break it down and pick up some important parts in this talk to make the essential parts clear.

Pyodide project is being improved with new browser features and these problems should be removed when WebAssembly JavaScript Promise Integration (JSPI) is available, but these contents will still be useful to understand the practical use cases of the AST manipulation.