We are using Amazon EBS to store a large number of small files (<10KB) in a 3-level directory structure.
~/lists# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 3.9G 5.5G 42% / tmpfs 854M 0 854M 0% /lib/init/rw varrun 854M 64K 854M 1% /var/run varlock 854M 0 854M 0% /var/lock udev 854M 80K 854M 1% /dev tmpfs 854M 0 854M 0% /dev/shm /dev/sda2 147G 80G 60G 58% /mnt /dev/sdj 197G 60G 128G 32% /vol
The partition in question is /vol (size: 200GB)
~/lists# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 655360 26541 628819 5% / tmpfs 186059 3 186056 1% /lib/init/rw varrun 186059 31 186028 1% /var/run varlock 186059 2 186057 1% /var/lock udev 186059 824 185235 1% /dev tmpfs 186059 1 186058 1% /dev/shm /dev/sda2 19546112 17573097 1973015 90% /mnt /dev/sdj 13107200 13107200 0 100% /vol
~/lists# sudo /sbin/dumpe2fs /dev/sdj | grep "Block size" dumpe2fs 1.41.4 (27-Jan-2009) Block size: 4096
The number of inodes for the partition /vol are 13Million+. The block size is 4096. Taking the Block Size as 4096, the number of inodes the 200GB partition (ext3) should support is 52million+ (Maximum Inode Calculation: Volume size in bytes/2^12). So why does the partition only support 13million inode?
I'm pretty sure that inodes are allocated statically when you create the volume (using mfs.ext3 in this case). For whatever reason, mkfs.ext3 decided to reserve 13 Million inodes and now you can't create any more files.
See this 2001 discussion of inodes
The Wikipedia ext3 page has a footnote explaining this more concisely: wiki link
Also, inodes are allocated per file (not block), which is why there are only 13M inodes - mkfs.ext3 must have been configured with an average file size of 8 KB which would account for the issue you're seeing.