git: fatal: could not find pack 'pack-xxxxxxx.pack'

Having a strange problem. Not sure how I caused it.

$ git gc
Enumerating objects: 625644, done.
Counting objects: 100% (625644/625644), done.
Delta compression using up to 16 threads
Compressing objects: 100% (126399/126399), done.
Writing objects: 100% (625644/625644), done.
Total 625644 (delta 497529), reused 622563 (delta 494488), pack-reused 0
fatal: could not find pack 'pack-6f30656f301f5f88438a5216f1df773bafcdf6d3.pack'
fatal: failed to run repack

I can see the file it's referring to, not sure if it's corrupted or not. It's all binary.

$ ls -lsah .git/objects/pack/pack-6f30656f301f5f88438a5216f1df773bafcdf6d3.pack
1168 -rw-------  1 javier  staff   584K Oct  8  2021 .git/objects/pack/pack-6f30656f301f5f88438a5216f1df773bafcdf6d3.pack

Looking a little closer, this hash doesn't have an accompanying .rev/.idx like the rest of the .pack files have in this directory. Is that the issue?

That seems to be addressed with Git 2.42 (Q3 2023), which now avoids breakage of "git pack-objects --cruft"^(man) due to inconsistency between the way the code enumerates packfiles in the repository.

See commit 73320e4 (07 Jun 2023) by Taylor Blau (ttaylorr).
^{(Merged by Junio C Hamano -- gitster -- in commit e224f26, 26 Jun 2023)}

> ## builtin/repack.c: only collect fully-formed packs
> ^{Reported-by: Michael Haggerty}
> ^{Signed-off-by: Taylor Blau}

> To partition the set of packs based on which ones are "kept" (either they have a .keep file, or were otherwise marked via the --keep-pack option) and "non-kept" ones (anything else), git repack^(man) uses its collect_pack_filenames() function.
>
> Ordinarily, we would rely on a convenience function such as get_all_packs() to enumerate and partition the set of packs.
> But collect_pack_filenames() uses readdir() directly to read the contents of the "$GIT_DIR/objects/pack" directory, and adds each entry ending in ".pack" to the appropriate list (either kept, or non-kept as above).
>
> This is subtly racy, since collect_pack_filenames() may see a pack that is not fully staged (i.e., it is missing its ".idx" file).
> Ordinarily, this doesn't cause a problem.
> But it can cause issues when generating a cruft pack.
>
> This is because git repack feeds (among other things) the list of existing kept packs down to git pack-objects --cruft^(man) to indicate that any kept packs will not be removed from the repository (so that the cruft pack machinery can avoid packing objects that appear in those packs as cruft).
>
> But read_cruft_objects() lists packfiles by calling get_all_packs().
> So if a ".pack" file exists (necessary to get that pack to appear to collect_pack_filenames()), but doesn't have a corresponding ".idx" file (necessary to get that pack to appear via get_all_packs()), we'll complain with:
>
> fatal: could not find pack '.tmp-5841-pack-a6b0150558609c323c496ced21de6f4b66589260.pack'
>
> Fix the above by teaching collect_pack_filenames() to only collect packs with their corresponding *.idx files in place, indicating that those packs have been fully staged.
>
> There are a couple of things worth noting:
>
> - Since each entry in the extra_keep list (which contains the
> --keep-pack names) has a *.pack suffix, we'll have to swap the
> suffix from ".pack" to ".idx", and compare that instead.
>
> - Since we use the the fname_kept_list to figure out which packs to delete (with git repack -d^(man)), we would have previously deleted a
> *.pack with no index (since the existince of a ".pack" file is
> necessary and sufficient to include that pack in the list of
> existing non-kept packs).
>
> Now we will leave it alone (since that pack won't appear in the
> list). This is far more correct behavior, since we don't want
> to race with a pack being staged. Deleting a partially staged pack
> is unlikely, however, since the window of time between staging a
> pack and moving its .idx file into place is miniscule.
>
> Note that this window does not include the time it takes to
> receive and index the pack, since the incoming data goes into
> "$GIT_DIR/objects/tmp_pack_XXXXXX", which does not end in ".pack"
> and is thus ignored by collect_pack_filenames().
>
> In the future, this function should probably be rewritten as a callback to for_each_file_in_pack_dir(), but this is the simplest change we could do in the short-term.

And, still with Git 2.42 (Q3 2023), we create .pack and then .idx, we consider only packfiles that have .idx usable (those with only .pack are not ready yet), so we should remove .idx before removing .pack for consistency.

See commit 0dd1324 (20 Jun 2023) by Derrick Stolee (derrickstolee).
^{(Merged by Junio C Hamano -- gitster -- in commit b2166b0, 29 Jun 2023)}

> ## packfile: delete .idx files before .pack files
> ^{Signed-off-by: Derrick Stolee}

> When installing a packfile, we place the .pack file before the .idx file.
> The intention is that Git scans for .idx files in the pack directory and then loads the .pack files from that list.
>
> However, when we delete packfiles, we do not do this in the reverse order as we should.
> The unlink_pack_path() method deletes the .pack followed by the .idx.
>
> This creates a window where the process could be interrupted between the .pack deletion and the .idx deletion, leaving the repository in a state that looks strange, but isn't actually too problematic if we assume the pack was safe to delete.
> The .idx without a .pack will cause some overhead, but will not interrupt other Git processes.
>
> This ordering was introduced into the 'git repack'^(man) builtin by a1bbc6c ("repack: rewrite the shell script in C", 2013-09-15, Git v1.8.5-rc0 -- merge), though we must be careful to track history through the code move in 8434e85 ("repack: refactor pack deletion for future use", 2019-06-10, Git v2.23.0-rc0 -- merge listed in batch #6) to see that.
>
> This became more important after 73320e4 ("builtin/repack.c: only collect fully-formed packs", 2023-06-07, Git v2.42.0 -- merge listed in batch #5) changed how 'git repack' scanned for packfiles for use in the cruft pack process.
> It previously looked for .pack files, but that was problematic due to the order that packs are installed: repacks between the creation of a .pack and the creation of its .idx would result in hard failures.
>
> There is an independent proposal about what to do in the case of a .idx without a .pack during this 'git repack' scenario, but this change is focused on deleting .pack files more safely.
>
> Modify the order to delete the .idx before the .pack.
> The rest of the modifiers on the .pack should still come after the .pack so we know all of the presumed properties of the packfile as long as it exists in the filesystem, in case we wish to reinstate it by re-indexing the .pack file.

The recent change to "git repack"^(man) made it react less nicely when a leftover .idx file that no longer has the corresponding .pack file in the repository: that has been corrected with Git 2.42 (Q3 2023).

See commit def390d (11 Jul 2023) by Taylor Blau (ttaylorr).
See commit 0af0672 (11 Jul 2023) by Derrick Stolee (derrickstolee).
^{(Merged by Junio C Hamano -- gitster -- in commit c6a5e1a, 18 Jul 2023)}

> ## builtin/repack.c: avoid dir traversal in collect_pack_filenames()
> ^{Signed-off-by: Taylor Blau}

> When repacking, the function collect_pack_filenames() is responsible for collecting the set of existing packs in the repository, and partitioning them into "kept" (if the pack has a ".keep" file or was given via --keep-pack) and "nonkept" (otherwise) lists.
>
> This function comes from the original C port of git-repack.sh from back in a1bbc6c ("repack: rewrite the shell script in C", 2013-09-15, Git v1.8.5-rc0 -- merge), where it first appears as get_non_kept_pack_filenames().
> At the time, the implementation was a fairly direct translation from the relevant portion of git-repack.sh, which looped over the results of
>
> find "$PACKDIR" -type f -name '*.pack'
>
> either ignoring the pack as kept, or adding it to the list of existing packs.
>
> So the choice to directly translate this function in terms of readdir() in a1bbc6c made sense.
> At the time, it was possible to refine the C version in terms of packed_git structs, but was never done.
>
> However, manually enumerating a repository's packs via readdir() is confusing and error-prone.
> It leads to frustrating inconsistencies between which packs Git considers to be part of a repository (i.e., could be found in the list of packs from get_all_packs()), and which packs collect_pack_filenames() considers to meet the same criteria.
>
> This bit us in 73320e4 ("builtin/repack.c: only collect fully-formed packs", 2023-06-07, Git v2.42.0 -- merge listed in batch #5), and again in the previous commit.
>
> Prevent these issues from biting us in the future by implementing the collect_pack_filenames() function by looping over an array of pointers to packed_git structs, ensuring that we use the same criteria to determine the set of available packs.
>
> One gotcha here is that we have to ignore non-local packs, since the original version of collect_pack_filenames() only looks at the local pack directory to collect existing packs.