put this line along with setting recyclerView. issue was fixed by
setting ItemAnimator to null for RecyclerView.
in kotlin
recyclerView.layoutManager = LinearLayoutManager(this)
recyclerView.itemAnimator = null
in java
recyclerView.setItemAnimator(null);