You should read more about how git-annex actually works. It doesn't track the files in git; it tracks the presence and location of the files, and their hashes. git-annex transfers the actual files around separately, tracks what repositories have them, and ensures that enough redundant copies of those files exist.