I’ve been playing with tokens for my application as well. While I’m not an expert by any means, I can share some of my experiences and thoughts on the matter.
The point of JWTs is essentially integrity. It provides a mechanism for your server verify that the token that was provided to it is genuine and was supplied by your server. The signature generated via your secret is what provides for this. So, yes, if your secret is leaked somehow, that individual can generate tokens that your server would think are its own. A token based system would still be more secure than your username/password system simply because of the signature verification. And in this case, if someone has your secret anyway, your system has other security issues to deal with than someone making fake tokens (and even then, just changing the secret ensures that any tokens made with the old secret are now invalid).
As for payload, the signature will only tell you that the token provided to you was exactly as it was when your server sent it out. verifying the that the payloads contents are valid or appropriate for your application is obviously up to you.
For your questions:
1.) In my limited experience, it’s definitely better to verify your tokens with a second system. Simply validating the signature just means that the token was generated with your secret. Storing any created tokens in some sort of DB (redis, memcache/sql/mongo, or some other storage) is a fantastic way of assuring that you only accept tokens that your server has created. In this scenario, even if your secret is leaked, it won’t matter too much as any generated tokens won’t be valid anyway. This is the approach I’m taking with my system – all generated tokens are stored in a DB (redis) and on each request, I verify that the token is in my DB before I accept it. This way tokens can be revoked for any reason, such as tokens that were released into the wild somehow, user logout, password changes, secret changes, etc.
2.) This is something I don’t have much experience in and is something I’m still actively researching as I’m not a security professional. If you find any resources, feel free to post them here! Currently, I’m just using a private key that I load from disk, but obviously that is far from the best or most secure solution.