This really piqued my interest. I was able to replicate your findings in my lab with the same pattern of results that you describe. I used Procmon to to try to see what changes are made and almost gave up until I saw the following:
That shows lsass.exe (Local Security Authority) writing to the local SAM and making a change(s) to the built-in Guest account (well-known RID 501). Sure enough, when I retested your scenario while watching the Guest account’s status, I see it enabled when “Password protected sharing” is disabled. However, when “Password protected sharing” is re-enabled, the guest account is not disabled again. Manually disable the guest account restores the original functionality: I am prompted for credentials (i.e. your third case).
I’m not sure why this behaves like this. TO be honest, I’d never even toggled the “Password protected sharing” setting before today (or even noticed it, for that matter). I hope this helps with your project. If someone else is interested in digging further, it would be interesting to know if this behavior is still present on Server 2012/2012 R2…
Oh and to your original questions (Is this by design or is it a bug?), I haven’t the slightest idea…