Opening a file with the encoding set to utf-8, and then passing that file to the .to_json function fixes the problem:
with open('df.json', 'w', encoding='utf-8') as file:
df.to_json(file, force_ascii=False)
gives the correct:
{"0":{"0":"τ","1":"π"},"1":{"0":"a","1":"b"},"2":{"0":1,"1":2}}
Note: it does still require the force_ascii=False argument.