I found I can do this by editing launchSettings.json and specifying the values there. This is similar to Arjun’s answer, but as far as I can tell there’s no UI to add the port numbers like when selecting IIS Express – so you have to edit the file directly.
Update: Actually, in my case, I still had problems getting the port numbers to be consistent after doing this. What worked for me was right-clicking on the project and adding “Container Orchestration Support”. Then set the port numbers in the docker-compose project. This helped with overriding environment variables as well.
Update 2: For anyone changing the port numbers in launchSettings.json, you may also need to delete any existing containers and/or images for the changes to take effect. Cleaning the solution wasn’t sufficient for me. I had to delete them from Docker Desktop.