Fragmentation

@echo off
setlocal
::Weekly defrag script; will defrag all locally attached hard disk partitions
set wmiccommand=wmic logicaldisk where "Description='Local Fixed Disk'" get caption
for /f "skip=1" %%a in ('%wmiccommand%') do call :DRIVECOMMANDS %%a
endlocal
goto :eof

:DRIVECOMMANDS
::each command here will be run against every local fixed 
::disk (hard drive letter, C:, E:, etc) on the system
::CDroms, floppies, removable disks, network drives will NOT be included

::the driveletter is represented as %1
defrag %1

schtasks /create /tn WeeklyDefrag /tr c:\batch\weeklydfrag.cmd /sc WEEKLY /mo 1 /d SAT /st 03:30:00 ru "System"

---- Everything below here is more relevant to XP, 2003, and older versions of Windows. ----

WindowsITPro does a study of The Impact of Disk Fragmentation

IDC study of fragmentation on NT/2000 (note old system specs) has NSTL results showing up to 80% performance degredation

DisKeeper documents How NTFS Reads a File. However they fail to note that often much of the MFT is cached in memory.

DisKeeper's Book on Disk Fragmentation

NTFS Optimization at NTFS.com

Some interesting notes on the defragger in Vista, like: why no more progress bar? Why does it seem less efficient?

JKdefrag uses an interesting strategy, and is free. I have not yet tried it. Mind the 'Known Problems' section near bottom of page!

NSTL Whitepaper (NT, 1999) showed an Excel performance degradation (operations took more than 2x as long!) at only 13% file fragmentation.

Michael Kessler of MS notes that defrag.exe cannot defrag "the MFT, the Paging File, FAT directories, or files open for exclusive use—for example, Windows registry." This is corroborated in KB227350 (for Windows 2000 only), which also gives a workaround method for page file defragging, though PageDefrag is easier, and can defrag paging files, registry hives, eventlog files, and hibernation files. However, KB227463 documents "In Windows 2000, it does not defragment NTFS metadata files, such as the Master File Table (MFT), or the metadata that describes a directory's contents. This limitation has been removed in Windows XP and later. It cannot defragment encrypted files in Windows 2000. This limitation has been removed in Windows XP and later." 

I had some difficulty reconciling KB227463 with How Disk Defragmenter Works - specifically, it was difficult to determine whether MFT defragmentation appeared in W2003 or in XP. Luckily I found Windows XP: Kernel Improvements Create a More Robust, Powerful, and Scalable OS (scroll down to the section on Defragmentation Improvements), which confirms that this change appeared in XP.

Maintaining Windows 2000 Peak Performance Through Defragmentation - explains how the pre-Vista defrag tool works.

Experiments

I did a few experiments:

• On an XP system, defrag c: -a -v reported that the MFT was in 3 fragments. After defrag, still 3 fragments.
• On a W2003R2 system, defrag c: -a -v reported that the MFT was in two fragments. After a defrag,  the MFT was still in two fragments.
• On a Vista system, defrag c: -a -v reported that the MFT was in 27 fragments! After a defrag of the volume, there were only 3 fragments.

From this I surmise that 3 fragments is OK, and the defragger won't take action. 27 is way too many, and will be fixed. So, somewhere between 4 and 27 fragments is a bad thing.

Serdar Yegulalp at Techtarget says: "I would argue that fragmented free space really becomes critical only when free space on a hard disk drive becomes extremely low—i.e., when the only space available is badly fragmented free space, and the system is forced to create new files in a highly fragmented fashion. But on a large enough drive, where the free space isn't allowed to go below 30%, this should almost never be an issue. There may still be fragmentation of free space, but large enough blocks of free space will almost certainly always exist somewhere on the drive to ensure that files can be moved or newly allocated without trouble. "

Mark Patton (formerly of DisKeeper) says "Larger and faster drives have minimized the impact of fragmentation. The Windows file system tends to fragment files all on its own to a small degree, but fragmentation starts for real when the drive starts to get full—as in over 70%. As the drive fills up, the larger areas of free space become scarce and the file system has no choice but to splatter large files around the disk. As the drive gets really full (over 90%), the file system then starts to fragment the MFT and the Pagefile. Now you've got a full drive, with lots of fragmented files, making the job of the defragmenter nearly impossible because it also needs space to do its job. It is my opinion that a drive that is more than 80% full is not defragmentable. You can see that effect with huge hard disk drives, since they generally use smaller percentages of the drive's total free space. A side-effect is that the overall fragmentation is reduced, and the fact that these drives have faster seek times makes the effect even less noticeable. "

Mike Kronenberg, who wrote a defragger for OnTrack, says "I challenge any defrag company to prove that, on a modern 2006 large drive about 50% full, defragmenting files will increase performance in any way that will be sensed by a user. Basically, nowadays defragmenting files will only provide a moderate performance boost when a drive is relatively full. Modern computers come with 250GB and larger drives that most people will never fill up." Additionally he noted that users are unable to tell the difference between Word loading in 6 seconds or 6.2 seconds.

A Diskeeper whitepaper on Windows2003/XP fragmentation states: "The XP/2003 NTFS file system driver maintains a list of the largest free spaces on the volume." ... "When a file gets created, it gets created in the free space that most closely matches the size of data available to write, in other words a "best fit". Additionally, a presumption is made that a newly created file will end up larger than the size that is currently available for the operating system to write, and extra free space, an “overallocation”, is reserved for the file so as to prevent the file from fragmenting (see Microsoft Knowledge Base article Q228198). That presumption is that the file will be 2, 4, 8 or 16 times larger than the currently known data size, depending on how much data is currently available for writing to the file in the operating system’s file cache."

In MFT Fragmentation, Lance Jensen points out the obvious: "The Master File Table (MFT) is the heart of the NTFS file system. It is essentially an index to all of the files on an NTFS volume, containing the file name, a list of the file attributes, and pointers to the fragments." ... "Fragmentation of the MFT can be a problem on NTFS partitions. This is because the MFT is used for every disk I/O. While much of the MFT can be cached so that an actual disk I/O does not have to be performed every time, it is still true that on most systems the MFT is accessed more than any other file. This means that MFT fragmentation is likely to have more impact on the system than fragmentation of any other single file."

Placed in context with The Four Stages of NTFS File Growth, the above statement gets more interesting!

Linux

Occasionally an argument about fragmentation on Linux filesystems vs NTFS will break out. A few linux-isms:

• ext2/ext3
  • The general consensus is that fragmentation doesn't become an issue until the disk is nearly full.
  • There really isn't a good tool for diagnosing fragmentation, or for defragging ext2/ext3 filesystems. Although you can get a percent fragmentation number with fsck or e2fsck, you must unmount the filesystem to do so correction: you can use e2fsck -n (thanks Kevin!). Honestly I don't know which is more true:
    • No one measures fragmentation on ext2/ext3 because fragmentation isn't a problem, or
    • Fragmentation on ext2/ext3 isn't considered a problem because no one ever measures it!
  • ext2, according to Daniel Robbins, Chief Architect of the Gentoo Project, "does not get fragmented easily, fragmentation is a one-way, cumulative process. That is, while ext2 fragments slowly, it cannot defragment itself. In other words, any often-modified ext2 filesystem will gradually get more and more fragmented, and thus slower. Even worse, there are no production-quality ext2 filesystem defragmenting programs currently available."
  • ext2's main defense against fragmentation is preallocation - when writing a new file, it allocates extra blocks so the file will have room to grow. A good explanation of it can be found here, from an author who believes defragmentation on ext2/ext3 is silly.
  • Generally speaking, the way to defrag an ext2/ext3 system is to back it up, destry the prime copy, then restore from backup.
• ReiserFS
  • Uses ext2 preallocation strategy to resist fragmentation (according to Hans Reiser in 2001)
  • No tools to measure fragmentation, or to defragment.
• XFS
  • Uses delayed allocation, space preallocation and space coallescing on deletion to avoid fragmentation (according to Nathan Scott in 2001)
  • Does have a defrag tool, xfs_fsr
• JFS
  • uses extents (basically another form of preallocation) to stave off fragmentation (according to Steve Best in 2001)
  • does have a defrag tool, defragfs. Can measure fragmentation with defragfs -q/-r

Constantin Loizides did some experiments to judge fragmentation among multiple filesystems. See the link, but basically after giving several filesystems a workout that should fragment them quite a bit, he observed the following fragmentation levels (although it is worthwhile to read his definition of fragmentation before trying to interpret this table):