Ideally either I’d like nginx to not serve https at all unless the hostname matches, or for it to redirect to http at the same host.
Neither is possible. The connection from a client that goes to https://foo.example.com/ cannot be accepted by anything but an SSL certificate with “foo.example.com” as one of its names. There is no opportunity to redirect until the SSL connection is accepted.
If you configure each site for SSL, a user who clicks through the certificate error will get the site they requested. If you configure a “catch all” site for SSL that provides only an error page and configure name-based virtual hosting for the one site that is supposed to support SSL, you can serve an error page to clients.
SSL and HTTP virtual hosting just don’t play nicely together.