So what it is gained from specializing std::pair?
Performance. The generic swap is usually good enough (since C++11), but rarely optimal (for std::pair, and for most other data structures).
I’m also left wondering if I should be writing my own specializations for custom classes, or simply relying on the template version.
I suggest relying on the template by default, but if profiling shows it to be a bottleneck, know that there is probably room for improvement. Premature optimization and all that…