Doing an rm -rf on a massive directory tree takes hours

No. rm -rf does a recursive depth-first traversal of your filesystem, calling unlink() on every file. The two operations that cause the process to go slowly are opendir()/readdir() and unlink(). opendir() and readdir() are dependent on the number of files in the directory. unlink() is dependent on the size of the file being deleted. The … Read more

Do I need to defrag unix filesystems?

Quoting Wikipedia’s page about the Ext3 Filesystem Modern Linux filesystem(s) keep fragmentation at a minimum by keeping all blocks in a file close together, even if they can’t be stored in consecutive sectors. Some filesystems, like ext3, effectively allocate the free block that is nearest to other blocks in a file. Therefore it is not … Read more

How many files can I put in a directory?

FAT32: Maximum number of files: 268,173,300 Maximum number of files per directory: 216 – 1 (65,535) Maximum file size: 2 GiB – 1 without LFS, 4 GiB – 1 with NTFS: Maximum number of files: 232 – 1 (4,294,967,295) Maximum file size Implementation: 244 – 26 bytes (16 TiB – 64 KiB) Theoretical: 264 – 26 bytes (16 EiB – 64 KiB) Maximum volume size Implementation: 232 – 1 clusters (256 TiB – 64 KiB) Theoretical: 264 – 1 clusters (1 YiB – 64 KiB) ext2: Maximum number of files: 1018 … Read more

ZFS vs XFS

I’ve found XFS more well suited to extremely large filesystems with possibly many large files. I’ve had a functioning 3.6TB XFS filesystem for over 2 years now with no problems. Definately works better than ext3, etc at that size (especially when dealing with many large files and lots of I/O). What you get with ZFS … Read more