What does the git index contain EXACTLY?

Bit by bit analysis

I've decided to do a little testing to better understand the format and research some of the fields in more detail.

Results bellow are the same for Git versions 1.8.5.2 and 2.3.

I have marked points which I'm not sure / haven't found with TODO: please feel free to complement those points.

As others mentioned, the index is stored under .git/index, not as a standard tree object, and its format is binary and documented at: https://github.com/git/git/blob/master/Documentation/technical/index-format.txt

The major structs that define the index are at cache.h, because the index is a cache for creating commits.

Setup

When we start a test repository with:

git init
echo a > b
git add b
tree --charset=ascii

The .git directory looks like:

.git/objects/
|-- 78
|   `-- 981922613b2afb6025042ff6bd878ac1994e85
|-- info
`-- pack

And if we get the content of the only object:

git cat-file -p 78981922613b2afb6025042ff6bd878ac1994e85

We get a. This indicates that:

  • the index points to the file contents, since git add b created a blob object
  • it stores the metadata in the index file, not in a tree object, since there was only a single object: the blob (on regular Git objects, blob metadata is stored on the tree)

hd analysis

Now let's look at the index itself:

hd .git/index

Gives:

00000000  44 49 52 43 00 00 00 02  00 00 00 01 54 09 76 e6  |DIRC.... ....T.v.|
00000010  1d 81 6f c6 54 09 76 e6  1d 81 6f c6 00 00 08 05  |..o.T.v. ..o.....|
00000020  00 e4 2e 76 00 00 81 a4  00 00 03 e8 00 00 03 e8  |...v.... ........|
00000030  00 00 00 02 78 98 19 22  61 3b 2a fb 60 25 04 2f  |....x.." a;*.`%./|
00000040  f6 bd 87 8a c1 99 4e 85  00 01 62 00 ee 33 c0 3a  |......N. ..b..3.:|
00000050  be 41 4b 1f d7 1d 33 a9  da d4 93 9a 09 ab 49 94  |.AK...3. ......I.|
00000060

Next we will conclude:

  | 0           | 4            | 8           | C              |
  |-------------|--------------|-------------|----------------|
0 | DIRC        | Version      | File count  | ctime       ...| 0
  | ...         | mtime                      | device         |
2 | inode       | mode         | UID         | GID            | 2
  | File size   | Entry SHA-1                              ...|
4 | ...                        | Flags       | Index SHA-1 ...| 4
  | ...                                                       |

First comes the header, defined at: struct cache_header:

  • 44 49 52 43: DIRC. TODO: why is this necessary?

  • 00 00 00 02: format version: 2. The index format has evolved with time. Currently there exists version up to 4. The format of the index should not be an issue when collaborating between different computers on GitHub because bare repositories don't store the index: it is generated at clone time.

  • 00 00 00 01: count of files on the index: just one, b.

Next starts a list of index entries, defined by struct cache_entry Here we have just one. It contains:

  • a bunch of file metadata: 8 byte ctime, 8 byte mtime, then 4 byte: device, inode, mode, UID and GID.

    Note how:

    • ctime and mtime are the same (54 09 76 e6 1d 81 6f c6) as expected since we haven't modified the file

      The first bytes are seconds since EPOCH in hex:

      date --date="@$(printf "%x" "540976e6")"

      Gives:

      Fri Sep  5 10:40:06 CEST 2014

      Which is when I made this example.

      The second 4 bytes are nanoseconds.

    • UID and GID are 00 00 03 e8, 1000 in hex: a common value for single user setups.

    All of this metadata, most of which is not present in tree objects, allows Git to check if a file has changed quickly without comparing the entire contents.

  • at the beginning of line 30: 00 00 00 02: file size: 2 bytes (a and \n from echo)

  • 78 98 19 22 ... c1 99 4e 85: 20 byte SHA-1 over the previous content of the entry. Note that according to my experiments with the assume valid flag, the flags that follow it are not considered in this SHA-1.

  • 2 byte flags: 00 01

    • 1 bit: assume valid flag. My investigations indicate that this poorly named flag is where git update-index --assume-unchanged stores its state: https://stackoverflow.com/a/28657085/895245

    • 1 bit extended flag. Determines if the extended flags are present or not. Must be 0 on version 2 which does not have extended flags.

    • 2 bit stage flag used during merge. Stages are documented in man git-merge:

      • 0: regular file, not in a merge conflict
      • 1: base
      • 2: ours
      • 3: theirs

      During a merge conflict, all stages from 1-3 are stored in the index to allow operations like git checkout --ours.

      If you git add, then a stage 0 is added to the index for the path, and Git will know that the conflict has been marked as solved. TODO: check this.

    • 12 bit length of the path that will follow: 0 01: 1 byte only since the path was b

  • 2 byte extended flags. Only meaningful if the "extended flag" was set on the basic flags. TODO.

  • 62 (ASCII b): variable length path. Length determined in the previous flags, here just 1 byte, b.

Then comes a 00: 1-8 bytes of zero padding so that the path will be null-terminated and the index will end in a multiple of 8 bytes. This only happens before index version 4.

No extensions were used. Git knows this because there would not be enough space left in the file for the checksum.

Finally there is a 20 byte checksum ee 33 c0 3a .. 09 ab 49 94 over the content of the index.


The Git book contains an article on what an index includes:

The index is a binary file (generally kept in .git/index) containing a sorted list of path names, each with permissions and the SHA1 of a blob object; git ls-files can show you the contents of the index:

$ git ls-files --stage
100644 63c918c667fa005ff12ad89437f2fdc80926e21c 0   .gitignore
100644 5529b198e8d14decbe4ad99db3f7fb632de0439d 0   .mailmap

The Racy git problem gives some more details on that structure:

The index is one of the most important data structures in git.
It represents a virtual working tree state by recording list of paths and their object names and serves as a staging area to write out the next tree object to be committed.
The state is "virtual" in the sense that it does not necessarily have to, and often does not, match the files in the working tree.

Nov. 2021: see also "Make your monorepo feel small with Git’s sparse index" from Derrick Stolee (Microsoft/GitHub)

https://github.blog/wp-content/uploads/2021/11/Fig-1-working-directory-index-commit-history.png

The Git index is a critical data structure in Git. It serves as the “staging area” between the files you have on your filesystem and your commit history.

  • When you run git add, the files from your working directory are hashed and stored as objects in the index, leading them to be “staged changes”.
  • When you run git commit, the staged changes as stored in the index are used to create that new commit.
  • When you run git checkout, Git takes the data from a commit and writes it to the working directory and the index.

In addition to storing your staged changes, the index also stores filesystem information about your working directory.
This helps Git report changed files more quickly.

To see more, cf. "git/git/Documentation/technical/index-format.txt":

The Git index file has the following format

All binary numbers are in network byte order.
Version 2 is described here unless stated otherwise.

  • A 12-byte header consisting of:
  • 4-byte signature:
    The signature is { 'D', 'I', 'R', 'C' } (stands for "dircache")
  • 4-byte version number:
    The current supported versions are 2, 3 and 4.
  • 32-bit number of index entries.
  • A number of sorted index entries.
  • Extensions:
    Extensions are identified by signature.
    Optional extensions can be ignored if Git does not understand them.
    Git currently supports cached tree and resolve undo extensions.
  • 4-byte extension signature. If the first byte is 'A'..'Z' the extension is optional and can be ignored.
  • 32-bit size of the extension
  • Extension data
  • 160-bit SHA-1 over the content of the index file before this checksum.

mljrg comments:

If the index is the place where the next commit is prepared, why doesn't "git ls-files -s" return nothing after commit?

Because the index represents what is being tracked, and right after a commit, what is being tracked is identical to the last commit (git diff --cached returns nothing).

So git ls-files -s lists all files tracked (object name, mode bits and stage number in the output).

That list (of element tracked) is initialized with the content of a commit.
When you switch branch, the index content is reset to the commit referenced by the branch you just switched to.

Git 2.20 (Q4 2018) adds an Index Entry Offset Table (IEOT):

See commit 77ff112, commit 3255089, commit abb4bb8, commit c780b9c, commit 3b1d9e0, commit 371ed0d (10 Oct 2018) by Ben Peart (benpeart).
See commit 252d079 (26 Sep 2018) by Nguyễn Thái Ngọc Duy (pclouds).
(Merged by Junio C Hamano -- gitster -- in commit e27bfaa, 19 Oct 2018)

ieot: add Index Entry Offset Table (IEOT) extension

This patch enables addressing the CPU cost of loading the index by adding additional data to the index that will allow us to efficiently multi- thread the loading and conversion of cache entries.

It accomplishes this by adding an (optional) index extension that is a table of offsets to blocks of cache entries in the index file.

To make this work for V4 indexes, when writing the cache entries, it periodically"resets" the prefix-compression by encoding the current entry as if the path name for the previous entry is completely different and saves the offset of that entry in the IEOT.
Basically, with V4 indexes, it generates offsets into blocks of prefix-compressed entries.

With the new index.threads config setting, the index loading is now faster.

As a result (of using IEOT), commit 7bd9631 clean-up the read-cache.c load_cache_entries_threaded() function for Git 2.23 (Q3 2019).

See commit 8373037, commit d713e88, commit d92349d, commit 113c29a, commit c95fc72, commit 7a2a721, commit c016579, commit be27fb7, commit 13a1781, commit 7bd9631, commit 3c1dce8, commit cf7a901, commit d64db5b, commit 76a7bc0 (09 May 2019) by Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit c0e78f7, 13 Jun 2019)

read-cache: drop unused parameter from threaded load

The load_cache_entries_threaded() function takes a src_offset parameter that it doesn't use. This has been there since its inception in 77ff112 (read-cache: load cache entries on worker threads, 2018-10-10, Git v2.20.0-rc0).

Digging on the mailing list, that parameter was part of an earlier iteration of the series, but became unnecessary when the code switched to using the IEOT extension.

With Git 2.29 (Q4 2020), the format description adjusts to the recent SHA-256 work.

See commit 8afa50a, commit 0756e61, commit 123712b, commit 5b6422a (15 Aug 2020) by Martin Ågren (none).
(Merged by Junio C Hamano -- gitster -- in commit 74a395c, 19 Aug 2020)

index-format.txt: document SHA-256 index format

Signed-off-by: Martin Ågren

Document that in SHA-1 repositories, we use SHA-1 and in SHA-256 repositories, we use SHA-256, then replace all other uses of "SHA-1" with something more neutral.
Avoid referring to "160-bit" hash values.

technical/index-format now includes in its man page:

All binary numbers are in network byte order.
In a repository using the traditional SHA-1, checksums and object IDs (object names) mentioned below are all computed using SHA-1.
Similarly, in SHA-256 repositories, these values are computed using SHA-256.

Version 2 is described here unless stated otherwise.


The Git index is a staging area between your working directory and your repository. You can use the index to build up a set of changes that you want to commit together. When you create a commit, what is committed is what is currently in this index, not what is in your working directory.

To see what is inside the index, issue the command:

git status

When you run git status, you can see which files are staged (currently in your index), which are modified but not yet staged, and which are completely untracked.

You can read this. A Google search throws up many links, which should be fairly self sufficient.

Tags:

Git

Internal

Recent Posts