I want to know what exact server in the end had the answer. Is that possible?
This is not achievable. You will not find the specific authoritative server that was consulted named anywhere in the payload of a DNS reply. There are CHAOS queries that exist for the purpose of identifying the specific recursive server that replied to you, but no such equivalent for extracting the name of the exact authoritative server that a recursor obtained the data from.
The StackOverflow Q&A that Phillip linked to provides instructions for how to identify the list of NS records that were present in the authority section of a recursive DNS response. This is the full list of servers though, not the specific server. Even were that sufficient for your uses, this information is considered optional by the DNS standards and it is increasingly rare to see it in recursive responses in order to reduce the payload size of internally sourced reflection attacks.
If you know the name of the zone, you can request the full list of NS records via an explicit request for them. This requires knowing that you are at the apex of the zone; a request for example.com NS records will return the desired response but www.example.com will not, unless www was delegated to another set of servers. In this case, you have no choice but to work backwards iteratively until you discover the apex of the zone.
Long story short, there is no reliable short path (i.e. single query) to getting information about all upstream authority from a recursive server, and there is no way to learn the specific authoritative server that responded to a query without also controlling the authoritative server and creating unique data on each server. (if you do this, please create a unique zone for that purpose so that it does not interfere with your more sensitive production zone transfers)