git rm documentation states:
--cachedis given, the staged content has to match either the tip of the branch or the file on disk, allowing the file to be removed from just the index.
So, for a single file:
git rm --cached file_to_remove.txt
and for a single directory:
git rm --cached -r directory_to_remove