What actually is "unallocated space"?
is it possible for other operating systems to write to unallocated space?
Yes, any OS can always write to any disk location, whether it's inside or outside a partition.
Partitions are nothing more than a list of regions that the OS agrees to use – the disk always behaves as a single continuous storage medium and doesn't know care about being partitioned.
(Indeed the partition table itself is stored on the same disk, on sectors 0–32ish, and when tools like fdisk or diskpart edit partitions, all they do is update the table stored in those sectors.)
does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?
It's not strictly necessary for most operating systems. A program with sufficient rights can open the raw device file such as /dev/sda4 on Linux or \\.\Device\Harddisk5\DR5 on Windows and read/write bytes into any location.
The program could practically invent its own private filesystem that way, or use some other structure (e.g. there are SQL database engines which write the DB into a raw partition). Or it could just dump data wherever.
Filesystems as you know them merely allow all programs within the OS to share the same drive or partition. The OS takes care of space allocation, reliability, etc. and programs don't need to think about anything other than files and folders.
When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?
When the "Disk Management" app prompts for the drive to be initialised, that means it does not even recognize the partition table (which is supposed to indicate which areas are allocated and which are not).
Windows understands MBR and GPT partition tables, but when this prompt shows up, it means neither was found. Therefore Windows doesn't even get to the point of looking at filesystems – it cannot know where the filesystems are (if there are any).
However, when Windows finds a partition with unrecognized filesystem, "Disk Management" does not prompt to format or otherwise overwrite it – it simply shows that partition as having an unrecognized filesystem, often shortened as "RAW".
When other operating systems create their partitions on an MBR or GPT disk, they will generally honor the existing partition table and won't start using a different one. So if e.g. Linux or FreeBSD created its own partitions, they will be visible as actual partitions in Disk Management – as long as they're in the MBR or GPT, they won't simply turn into "unallocated space" just because the filesystem was not recognized.
However, do not confuse DiskMgmt's prompt to initialize the disk with Windows Explorer's prompt to format the disk. The former is about unknown partition tables, but the latter is about unknown filesystems in a known partition.
(Sometimes Windows accidentally assigns a drive letter to an unknown filesystem, and trying to open it via Windows Explorer will cause a "Format" prompt.)
So in short:
Prompt to initialize disk ⇒ No partition table (MBR or GPT) was found. The disk might be empty or it might use a non-standard partition table such as APM.
No prompt, but unallocated space shown ⇒ The space is really unallocated. The disk's partition table has no entries that would correspond to that area.
No prompt, but unknown partition shown ⇒ The partition contains an unrecognized filesystem.
Prompt to format disk ⇒ The partition contains an unrecognized filesystem.
Let's first explain how a harddrive works.
A harddrive has space, obviously. You create a partition scheme across the entire drive, and you fill that with partitions. A partition is a reservation in the partition space with a fixed size. (this is a very basic explanation. If I add RAID to the mix, I fear the answer gets unnecessary complex)
If your harddrive has just one partition, and its size is smaller than the size of the harddisk, there is space left for another partition. This bit is called unallocated space.
Now, does this mean you can always create a partition there? No. It is possible that another OS created a partition in that space that is completely unfamiliar by your OS.
If this is the case, your OS will either see this as an unknown partition, or in the worst case as unallocated space. If you create a partition in unallocated space that is actually allocated by another OS, you will destroy the previous partition in most cases.
There is another possibility, the drive has a partition scheme that is not familiar by your OS. If this is the case, the OS will think the drive is not initialized, and will ask you to initialize it. In Windows, it will either be GPT or MBR, but there are known cases where the drive (or SD Card etc) is formatted for a special device and its unique format is a form of security.
A hard drive (or its equivalent such as a USB thumb drive) by convention has a partition table on it. An MBR (master boot record) style partition table fits in one sector (512 bytes) at the beginning of the drive (sector number 0). An MBR also contains a boot loader so the partition table space is limited to four "primary" partitions (though some 1980s software supported eight partitions). Each MBR partition entry has:
- a type code (a single byte) indicating that the corresponding partition is either NTFS, FAT-32 (four flavors), FAT-16 (eight flavors), FAT-12, Linux swap area, etc. Partition code 0 (zero) means "raw", or "no meaningful content", "or disregard partition contents".
- whether the partition is flagged as bootable. Up to one partition per disk can have this flag enabled.
- The disk address of the beginning sector of the partition
- The partition's ending sector—or the number of sectors. Early PCs used cylinder/head/sector numbers; after about 1995, use of "logical block numbers" a single 32-bit value replaced cyl/head/sect addressing
Since a maximum of four partitions was soon intolerable, a mechanism for additional partitions was "extended partitions" in MBR where a specially marked partition can have an embedded MBR containing another partition table. Thus it is possible to divide a disk into dozens of partitions though some partitioning tools might inflict some artificial limit.
A GPT is intrinsically not limited to four partitions. Its basics occupy more than 512 bytes on disk, usually something in the 4 to 16 megabyte range. But the meaning of partitions is the same.
A disk's partition table which has every sector of the disk included in it has "no unallocated space" by the most common use of the phrase. So the usual answer to your question is that unallocated space is that outside of all partitions. However, ambiguity of terminology occurs if a partition is marked as type zero (raw), so there possibly could exist a tool which considers a raw partition as, in some sense, unallocated.
When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?
It means there is no partition table or all the partition entries are empty. Windows is happy to ignore unrecognized filesystems.
Is the concept of unallocated space a Windows one, or a hard drive one - in other words, is it possible for other operating systems to write to unallocated space?
It is a hard drive convention observed by all standard operating systems supporting PC software. Not sure what "in other words" means. Yes, it is possible to write to unallocated space (even in Windows), but the proper etiquette is to create a partition containing the unallocated space, format a filesystem within it (which writes to the "unallocated space"), and then write files and directories to it.
If so, does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?
For a well behaved system, no. Cooperative systems require filesystems to be in place, permissions set appropriately, etc. However, it would be trivial in Linux (for example) to open the disk itself as a big file (raw disk access) and write directly to it. It would require administrative access and some guidance or calculations to determine where to write.
However, ignoring filesystems and partitions is bad. They greatly improve disk management and make life much easier and more sane. Ultimately it is possible to write renegade software which completely ignores sanity and good manners and writes anywhere on the disk it wants.
However, the meta-ness I perceive from this question might be misunderstanding whether you intend to obey the permissions and limits imposed by non-administrator/non-system mode in Windows, Linux, etc.