There is no problem. Even if there is still available RAM, the Linux Kernel will move memory pages which are hardly ever used into swap space.
It’s good to swap out memory pages that have been inactive for a while, keeping often-used data in cache; this is the desired situation of the Kernel.
You can have more control on this by using vfs_cache_pressure (which controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects) and swappiness (which controls how aggressive the kernel will swap memory pages: higher values will increase aggressiveness, lower values decrease the amount of swap).
You can find these settings in /etc/sysctl.conf.