This is git-annex's bug list. Link bugs to done when done.

subtle build issue on OSX 10.7 and Haskell Platform (if you have the 32bit version installed)
Posted Sat Sep 22 03:36:59 2012

watcher commits unlocked files
Posted Sat Sep 22 03:36:59 2012

unannex command doesn't all files
Posted Sat Sep 22 03:36:59 2012

Issue on OSX with some system limits
Posted Wed Sep 19 16:46:44 2012

annexed symlink mtime matching code is disabled on non-linux systems; needs testing
Posted Tue Jul 17 17:54:56 2012

unannex and uninit do not work when git index is broken
Posted Tue Jul 17 17:54:56 2012

Unfortunate interaction with Calibre
Posted Tue Jul 17 17:54:56 2012

Prevent accidental merges
Posted Tue Jul 17 17:54:56 2012

softlink mtime
Posted Tue Jul 17 17:54:56 2012

signal weirdness
Posted Tue Jul 17 17:54:56 2012

git rename detection on file move
Posted Tue Jul 17 17:54:56 2012

S3 memory leaks
Posted Tue Jul 17 17:54:56 2012

[[!edittemplate template=templates/bugtemplate match="bugs/*" silent=yes]]

The key is the basename of the symlink target.
Comment by http://joey.kitenet.net/ Sun May 15 16:47:53 2011
Maybe I will run into issues myself somewhere down the road, but generally speaking, I really really like the fact that files are immutable by default.
I've been trying to dig around the trace and code, and used google to see if the forkProcess issue was a haskell thing or an OSX thing. It seems that http://hackage.haskell.org/trac/ghc/ticket/4493 someone may have ran into a similar issue, though I am not sure if its related.

To re-inject new content for a file, you really want to get a new key for the file. Otherwise, other repos that have the old file will never get the new content. So:

git rm file
mv ~/newcontent file
git annex add file
Comment by http://joey.kitenet.net/ Sat May 14 16:28:36 2011

a0826293 fixed the last problem, there is coreutils available in macports, if they are installed you get the gnu equivalents but they are prefixed with a g (e.g. gchmod instead of chmod), I guess not everyone will have these install or prefer these on OSX

Some more tests fail now...

Testing 1:blackbox:3:git-annex unannex:1:with content
### Failure in: 1:blackbox:3:git-annex unannex:1:with content
foo is not a symlink
Testing 1:blackbox:4:git-annex drop:0:no remotes
### Failure in: 1:blackbox:4:git-annex drop:0:no remotes
drop wrongly succeeded with no known copy of file
Testing 1:blackbox:4:git-annex drop:1:with remote
Testing 1:blackbox:4:git-annex drop:2:untrusted remote
Testing 1:blackbox:5:git-annex get
Testing 1:blackbox:6:git-annex move
Testing 1:blackbox:7:git-annex copy
### Failure in: 1:blackbox:7:git-annex copy
move --to of file already there failed
Testing 1:blackbox:8:git-annex unlock/lock
### Error in:   1:blackbox:8:git-annex unlock/lock
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:9:git-annex edit/commit:0
### Error in:   1:blackbox:9:git-annex edit/commit:0
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:9:git-annex edit/commit:1
### Error in:   1:blackbox:9:git-annex edit/commit:1
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:10:git-annex fix
### Error in:   1:blackbox:10:git-annex fix
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:11:git-annex trust/untrust/semitrust
### Error in:   1:blackbox:11:git-annex trust/untrust/semitrust
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:12:git-annex fsck:0
### Error in:   1:blackbox:12:git-annex fsck:0
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:12:git-annex fsck:1
### Error in:   1:blackbox:12:git-annex fsck:1
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:12:git-annex fsck:2
### Error in:   1:blackbox:12:git-annex fsck:2
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:13:git-annex migrate:0
### Error in:   1:blackbox:13:git-annex migrate:0
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:13:git-annex migrate:1
### Error in:   1:blackbox:13:git-annex migrate:1
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:14:git-annex unused/dropunused
### Error in:   1:blackbox:14:git-annex unused/dropunused
forkProcess: resource exhausted (Resource temporarily unavailable)
Cases: 30  Tried: 30  Errors: 11  Failures: 3
test: failed
make: *** [test] Error 1

On a side note, I think I found another bug in the testing. I had tested in a virtual machine in archlinux (a very recent updated version) Please see the report here tests fail when there is no global .gitconfig for the user

Ah, great, thanks very much for the quick fix!

Yes, when I mentioned three defunct git processes, there were three processes shown as "git [defunct]", plus the three git processes I listed, plus two "git-annex" processes. Upon cancel/resume, there were no defunct git processes when I checked, but by the time I found the bug report on the forum and commented I'd already successfully upgraded by annex (by repeatedly attaching strace) and couldn't really easily get at either additional 'ps' info or a fuller strace than what I posted (that was just the log from one of the attach/detach cycles), so it's a relief you managed to pinpoint the problem.

Comment by https://lithitux.org/openidserver/users/pavel Wed Jul 6 08:14:26 2011
comment on the output of 'git-annex version' (from my last comment): now I get the right version 3.20110707. But I checked in my console that the three commands "git checkout 3.20110707", "make" and "./git-annex version" gave me before 3.20110702, I don't know why...

Lauri a scratch patch would be very helpful. Encoding stuff makes my head explode.

However, I am very worried by haskell's changes WRT unicode and filenames. Based on user input, git-annex users like to use it on diverse sets of files, with diverse and ill-defined encodings. Faffing about with converting between encodings seems likely to speactacularly fail.

Comment by http://joey.kitenet.net/ Sat Jan 28 19:40:34 2012

Or, even better, wouldn't it make sense to have SHA backends always default to --fast and only use non-fast when any snags are hit, use non-fast mode for that file.

Though if we continue here, we should probably move this to its own page.

Outside the test suite, git-annex's actual use of cp puts fairly low demands on it. It tries to use cp -a or cp -p if available just to preserve whatever attributes it can preserve, but the worst case if that you have a symlink pointing to a file that doesn't have the original timestamp or whatever. And there's little expectation git preserves that stuff anyway.

I will probably try to make the test suite entirely use git clone rather than cp.

Comment by http://joey.kitenet.net/ Sun Feb 13 17:54:09 2011

Joey, sorry, I got it wrong. I thought upgrading git didn't help and you adjusted things in git-annex instead.

Anyway, can I get around upgrading on all hosts by reformatting the drive to case-sensitive HFS+? Or will I have to upgrade git (currently version 1.7.2.5) eventually anyway?

Comment by gernot Sun Apr 3 19:46:16 2011

On second thought and after some messing (trying most of the options and combinations of options on OSX for).... I tried replacing cp with gnu cp from coreutils on my OSX install, and all the tests passed. sigh cp -a is preserving some permissions and attributes but not all, its not behaving in the same way as the gnu cp does... the closet thing that I have found on OSX that behaves in the same way as gnu "cp -pr" is to use "ditto".

Just doing a "ditto SOURCE DEST" in the tests passes everything. I'm not sure if its a good idea to use this even though it works. Though this is just the tests, does it affect CopyFile.hs where "cp" is called?

It seems the objects are in the remote after all, but the remote is unaware of this fact. No idea where/why the remote lost that info, but.. Anyway, with the SHA backends, wouldn't it make sense to simply return "OK" and update the annex logs accordingly, no?

Local:

% ls -l foo
lrwxrwxrwx 1 richih richih 312 Apr  3 01:18 foo -> .git/annex/objects/gG/VW/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491
% 

Remote:

% git-annex-shell recvkey <remote> SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491
git-annex-shell: key is already present in annex
% strace git-annex-shell recvkey /base/git-annex/fun SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491 2>&1 | grep SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491
stat64("/base/git-annex/fun/annex/objects/gG/VW/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491", {st_mode=S_IFREG|0444, st_size=80781, ...}) = 0
% ls -l /base/git-annex/fun/annex/objects/gG/VW/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491
-r--r--r-- 1 richih richih 80781 2011-04-01 12:44 /base/git-annex/fun/annex/objects/gG/VW/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491/SHA512-s80781--cef3966a19c7435acceb8fbfbff1feebe6decab7c81a0c197f00932cf9ef0eac330784cc3f0d211bd4acf56a6d16daaebe9b598aa4dfd5bfec73f4e6df3f0491
% 

So, there is evidence here of a circumstance caused by the other bug, as I suspected.

I don't think that manual git commit -a caused the problem. I suspect it was a subsequent git add that caused git to follow the wrong case paths and add the files in the wrong place. Ie, when you run "git add .git-annex", it recurses into .git-annex/Gm/, and adds files using that case, that were previously added from .git-annex/GM/.

For completeness, can you verify this repo's core.ignorecase setting?


I hate that you are stuck using loop filesystems to work around this bug. If my guess is correct, you don't need to, as long as you avoid manually running "git add .git-annex". I take this bug seriously. While I'm currently very involved in adding Amazon S3 support to git-annex (which will take days more of solid work), I do plan to make a loop filesystem of my own, probably vfat, so I can try and reproduce this on a case-insensative filesystem. If you could confirm my above hypothesis, that would speed things up for me.

It's possible I will have to tweak the hash directories. Hopefully if so, I will only tweak them for new keys; if I had to do a v3 backend just to fix this stupid thing, I'd be sad -- upgrading all my offline disks from v1 to v2 took me many days.

Comment by http://joey.kitenet.net/ Mon Mar 28 15:25:18 2011

I forgot to mention that the statfs64 stuff in OSX seems to be deprecated, see http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man2/statfs64.2.html

on a slightly different note, is anonymous pushing to the "wiki" over git allowed? I'd prefer to be able to edit stuff inline for updating some of my own comments if I can :P

I did a bit of research and my NAS had ancient NFS software on it. I upgraded that and things are now working as expected. Sorry for the noise.
Comment by http://cgray.myopenid.com/ Tue Nov 15 05:14:03 2011

Try the changes I've pushed to use statfs64 on apple.

There is actually a standardized statvfs that I'd rather use, but after the last time that I tried going with the POSIX option first only to find it was not broadly implemented, I was happy to find some already existing code that worked for some OSs.

(While ikiwiki supports anonymous git push, it's a feature we have not rolled out on Branchable.com yet, and anyway, ikiwiki disallows editing existing comments that way. I would, however, be happy to git pull changes from somewhere.)

Comment by http://joey.kitenet.net/ Wed Mar 23 16:57:56 2011
Just to make sure: How do I get $key? What I did was look at the path in the object store of the local repo and see if that exact same path & file existed in the remote.
oh, and that probably goes for 'help' and other subcommands as well.

That's odd, I have the md5sha1sum package installed and it still fails with pretty much the same error

Testing 1:blackbox:0:git-annex init
Cases: 30  Tried: 7  Errors: 0  Failures: 0chmod: -R: No such file or directory
### Error in:   1:blackbox:0:git-annex init
.t/repo/.git/annex/objects/SHA1:ee80d2cec57a3810db83b80e1b320df3a3721ffa/SHA1:ee80d2cec57a3810db83b80e1b320df3a3721ffa: removeLink: permission denied (Permission denied)
Testing 1:blackbox:1:git-annex add:0
### Error in:   1:blackbox:1:git-annex add:0
foo: openFile: permission denied (Permission denied)

< and so on >

the configure script finds sha1sum, builds and starts to run.

Fixed that, and removed the impossible cast so it can be built with #if 1
Comment by http://joey.kitenet.net/ Sun Mar 20 22:06:25 2011
And what about emitting a warning, as git does, that some files were not annex-added (when not using --force)?
For example if the file is owned by root, I guess git-annex fails when it tries to remove write permissions (I retested with the last version of today (whose "version" subcommand still outputs 3.20110702)).By the way, it would be nice to have a log file created containing the list of all failures, to avoid having to scan manually all the output of a long git-annex operation.

FYI, (the follow is on OSX 10.7 on two different machines)

On my 64bit install of haskell platform...

laplace:~ jtang$ ghc -e 'print System.Info.arch'
"x86_64"

On my 32bit install of haskell platform...

 x00:git-annex jtang$ ghc -e 'print System.Info.arch'
"i386"

Running cabal build or cabal install git-annex as you suggest with the 32bit install does do the right thing.

I have pushed out a preliminary fix. The old mixed-case directories will be left where they are, and still read from by git-annex. New data will be written to new, lower-case directories. I think that once git stops seeing changes being made to mixed-case, colliding directories, the bugs you ran into won't manifest any more.

You will need to find a way to get your git repository out of the state where it complains about uncommitted files (and won't let you commit them). I have not found a reliable way to do that; git reset --hard worked in one case but not in another. May need to clone a fresh git repository.

Let me know how it works out.

Comment by http://joey.kitenet.net/ Sat Apr 2 17:53:58 2011

What an evil little bug. In retrospect, this probably bit my own test upgrades, but I ran git annex fsck everywhere and so avoided the location log breakage.

I've fixed the bug, which also involved files with other punctuation in their names [&:%] when using the WORM backend.

The only way I have to recover repos that have already been upgraded is to run git annex fsck --fast in each clone of such a repo, which will let it rebuild the location log information. I think that is the best way to recover; ie I can't think of a way to recover that doesn't need to do everything fsck does anyway.

Comment by http://joey.kitenet.net/ Thu Jul 7 21:04:23 2011

So, it appears that you're using git annex copy --fast. As documented that assumes the location log is correct. So it avoids directly checking if the bare repo contains the file, and tries to upload it, and the bare repo is all like "but I've already got this file!". The only way to improve that behavior might be to let rsync go ahead and retransfer the file, which, with recovery, should require sending little data etc. But I can't say I like the idea much, as the repo already has the content, so unlocking it and letting rsync mess with it is an unnecessary risk. I think it's ok for --force to blow up if its assumptions turn out to be wrong.

If you use git annex copy without --fast in this situation, it will do the right thing.

Comment by http://joey.kitenet.net/ Sun May 15 19:40:47 2011

This only happens with the WORM backend (or possibly with SHA1E if the file's extension has a newline).

The problem is not the newline in the file, but the newline in the key generated for the file. It's probably best to just disallow such keys being created.

Comment by http://joey.kitenet.net/ Tue Dec 6 16:49:32 2011

Version: 0.20110503

My local non-bare repo is copying to a remote bare repo.

I have been recovering in a non-bare repo.

If there is anything I can send you to help... If I removed said files and went through http://git-annex.branchable.com/bugs/No_easy_way_to_re-inject_a_file_into_an_annex/ -- would that help?

In case this matters, I just realized that this disk has been formatted with NTFS instead of a sane FS.
Indeed, uninit needed to be improved. I've done so. Also, unannex --fast can be used to make hard links to content left in the annex.

In the meantime, would it be acceptable to split the pre-commit hook into two discrete parts?

This would allow to (if preferred) defer "git annex fix" until post-commit while still keeping the safety net for unlocked files.

Comment by praet Mon Mar 21 19:58:34 2011

Alternatively, you can just load it up in ghci and see if it reports numbers that make sense:

joey@gnu:~/src/git-annex>make StatFS.hs
hsc2hs StatFS.hsc
perl -i -pe 's/^{-# INCLUDE.*//' StatFS.hs
joey@gnu:~/src/git-annex>ghci StatFS.hs
GHCi, version 6.12.1: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling StatFS           ( StatFS.hs, interpreted )
Ok, modules loaded: StatFS.
*StatFS> s <- getFileSystemStats "."
Loading package bytestring-0.9.1.5 ... linking ... done.
*StatFS> s
Just (FileSystemStats {fsStatBlockSize = 4096, fsStatBlockCount = 7427989, fsStatByteCount = 30425042944, fsStatBytesFree = 2528489472, fsStatBytesAvailable = 2219384832, fsStatBytesUsed = 27896553472})
Comment by http://joey.kitenet.net/ Wed Mar 23 15:13:33 2011

Hi,

(I'm new to git and git annex, so please forgive any mistakes I make...)

My repo is messed up right now. The fact that I copied the repo with rsync -a back and forth from a case insensitive filesystem to a case sensitive one, probably didn't help.

I believe the annexed files in .git/annex/objects/ are still using a mixed case directory hashing scheme. That's the problem I'm having. The symlinks point to the wrong case and are now broken. I don't think the latest versions of git-annex changed that (it only changed the hashing under .git-annex, right?).

Even if I clean up my repo, I think I'm still going to have a problem because I have one repo on an OS X case insensitive filesystem and my other repos on case sensitive Linux filesystems. Potentially the directory name under .git/annex/objects will have a different case. Then the symlink might have a different case than my Linux FS. Does git-annex track changes in git by the contents of the symlink? In which case the case difference would show up as a change even though there is no change?

Is it possible to change the directory hashing scheme under .git/annex/objects to use lowercase names?

Comment by ssqq Thu Jun 2 20:31:55 2011

Seems like you probably have files in git with nearly as long filenames as the key files. Course, you can rename those yourself.

This couldn't be changed directly in WORM without some ugly transition, but it would be possible to implement it as a WORM100 or so. OTOH, if you're going to git annex migrate, you might as well use SHA1.

Comment by http://joey.kitenet.net/ Fri Apr 8 17:14:25 2011

A google search http://www.google.com/search?hl=en&sclient=psy-ab&q=conq%3A+invalid+command+syntax&btnG= finds other examples of this error message related to ssh, mercurial, and bitbucket. What that has to do with git is anyone's guess, but I'm pretty sure git-annex is not related to it at all.

Comment by http://joey.kitenet.net/ Tue Jan 3 00:41:08 2012

Hey @fmarier. Well, this bug report is closed because you can already get rid of the symlinks. Just put a bare git repo on your fat filesystem, and use git-annex copy --to/--from there.

Now, that puts all the files that are on the device in .git/annex/objects/xx/yy/blah.mp3 -- how well rockbox would support that I don't know. And if it tries to modify or delete those files, git annex also can't help you manage those changes.

Another recent option is the directory special remote type, which again uses "xx/yy/blah.mp3" and can't track changes made to the files. This could perhaps be extended in the direction you suggest, although trying to fit this into the special remote infrastructure might not be a good fit really.

The most likely way this has to get dealt with is really by using smudge filters, which would eliminate the symlinks and allow copying a non-bare git repo onto vfat.

Comment by http://joey.kitenet.net/ Mon Apr 4 18:20:45 2011

Yeap, that did the trick. I just tested a few separate OSX 10.6.6 systems and the tests are better behaved now, only 3 failures now.

So the tests behave better (at least we don't get resource fork errors any more)

  • after the commit c319a3 without modifying the system limits (of 266 procs per user)
  • without the commit c319a3 and when I increase the system process limits to as much as OSX allows

On all the systems I tested on, I'm down to 3 failures now.

### Failure in: 1:blackbox:3:git-annex unannex:1:with content
foo is not a symlink
### Failure in: 1:blackbox:4:git-annex drop:0:no remotes
drop wrongly succeeded with no known copy of file
Cases: 30  Tried: 20  Errors: 0  Failures: 2add foo ok
ok
Cases: 30  Tried: 24  Errors: 0  Failures: 2  Only 1 of 2 trustworthy copies of foo exist.
  Back it up with git-annex copy.
  Only 1 of 2 trustworthy copies of sha1foo exist.
  Back it up with git-annex copy.
  Bad file size; moved to /Users/jtang/develop/git-annex/.t/tmprepo/.git/annex/bad/WORM:1297594011:20:foo
  Bad file content; moved to /Users/jtang/develop/git-annex/.t/tmprepo/.git/annex/bad/SHA1:ee80d2cec57a3810db83b80e1b320df3a3721ffa
### Failure in: 1:blackbox:12:git-annex fsck:1
fsck failed to fail with content only available in untrusted (current) repository
Cases: 30  Tried: 26  Errors: 0  Failures: 3  Only 1 of 2 trustworthy copies of foo exist.
  Back it up with git-annex copy.
  The following untrusted locations may also have copies: 
    90d63906-375e-11e0-8867-abb8a6368269  -- test repo
  Only 1 of 2 trustworthy copies of sha1foo exist.
  Back it up with git-annex copy.
  The following untrusted locations may also have copies: 
    90d63906-375e-11e0-8867-abb8a6368269  -- test repo
Cases: 30  Tried: 30  Errors: 0  Failures: 3

It's the same set of failures across all the OSX systems that I have tested on. Now I just need to figure out why there are still these three failures.

Indeed, I've made it even more robust now, handling the case where the file has weird permissions too, and undoing the failed add so the file is always back at the start state. Had to add a dependency on another haskell module to allow this, so it took some time to figure out how to do it..
Comment by http://joey.kitenet.net/ Fri Jul 8 01:32:30 2011

Adam, this bug was fixed a long time ago, first using option #2 above, but later switching to option #3 -- git-annex treats filenames as opaque binary blobs and never decodes them in any encoding; haskell's normal encoding support for stdio is disabled.

And it never resulted in a failure like you show. I cannot reproduce your problem, but it is a different bug, please open a new bug report.

Comment by http://joey.kitenet.net/ Sat Dec 24 16:49:13 2011

That seems an excellent idea, also eliminating the need for git annex fix after moving.

However, I think CVS and svn have taught us the pain associated with a version control system putting something in every subdirectory. Would this pain be worth avoiding the minor pain of needing git annex fix and sometimes being unable to follow renames?

Comment by http://joey.kitenet.net/ Mon Dec 19 18:22:25 2011

It exists locally, whereis tells me it exists locally and locally, only.

The object is not in the bare repo.

The file might have gone missing before I upgraded my annex backend version to 2. Could this be a factor?

Won't git itself be fixed on this issue? It was on my plans to look into that, however I don't know how difficult it will be.

Hm, if path's ok, guess there's no way around git-bisect indeed. Wonder if there's some kind of ccache for haskell...

OS is linux, amd64 on "host1" and i386 on "host2" where git-annex-shell is crashing. I'll try to come up with a commit, thanks for clarifications.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Sun Apr 3 04:45:49 2011
The chmod errors are because your chmod does not understand the -R argument. Only the test suite uses chmod -R. I've fixed it to modify modes manually.
Comment by http://joey.kitenet.net/ Wed Feb 9 04:10:27 2011

Actually I may have just been stupid and should have read the man page on statfs...

jtang@x00:~/develop/git-annex $ git diff
diff --git a/StatFS.hsc b/StatFS.hsc
index 8b453dc..e10b2dd 100644
--- a/StatFS.hsc
+++ b/StatFS.hsc
@@ -53,7 +53,7 @@ import Foreign.C.String
 import Data.ByteString (useAsCString)
 import Data.ByteString.Char8 (pack)

-#if defined (__FreeBSD__)
+#if defined (__FreeBSD__) || defined (__APPLE__)
 # include 
 # include 
 #else
@@ -84,8 +84,8 @@ data CStatfs
 #ifdef UNKNOWN
 #warning free space checking code not available for this OS
 #else
-#if defined(__FreeBSD__)
-foreign import ccall unsafe "sys/mount.h statfs"
+#if defined(__FreeBSD__) || defined (__APPLE__)
+foreign import ccall unsafe "sys/mount.h statfs64"
 #else
 foreign import ccall unsafe "sys/vfs.h statfs64"
 #endif

yields this...

jtang@x00:~/develop/git-annex $ ghci StatFS.hs                                                                                                                                    
GHCi, version 6.12.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package ffi-1.0 ... linking ... done.
[1 of 1] Compiling StatFS           ( StatFS.hs, interpreted )
Ok, modules loaded: StatFS.
*StatFS> s <- getFileSystemStats "."
Loading package bytestring-0.9.1.7 ... linking ... done.
*StatFS> s
Just (FileSystemStats {fsStatBlockSize = 4096, fsStatBlockCount = 244106668, fsStatByteCount = 999860912128, fsStatBytesFree = 423097798656, fsStatBytesAvailable = 422835654656, fsStatBytesUsed = 576763113472})
*StatFS> 

we could just stick another if defined (APPLE) instead of what I previously had and it looks like it will do the right thing on OSX.

I also encountered Adam's bug. The problem seems to be that communication with the git process is done with Char8-bytestrings. So, when L.unpack is called, all filenames that git outputs (with ls-files or ls-tree) are interpreted to be in latin-1, which wreaks havoc if they are really in UTF-8.

I suspect that it would be enough to just switch to standard Strings (or Data.Text.Text) instead of bytestrings for textual data, and to Word8-bytestrings for pure binary data. GHC should nowadays handle locale-dependent encoding of Strings transparently.

Repeated bisect with -j1, just to be sure it's not a random error, and it gave me 828a84ba3341d4b7a84292d8b9002a8095dd2382 again. Guess I'll look through the changes there a bit later and try to revert these until it works.

Not sure if it's repeatable by anyone but me (and hence worth fixing), but here's a bit more of info about the system:

Exherbo linux
Linux sacrilege 2.6.38.2-fg.roam #4 SMP PREEMPT Mon Mar 28 21:08:47 YEKST 2011 i686 GNU/Linux

dev-lang/ghc-7.0.2:7.0.2::installed
dev-haskell/HUnit-1.2.2.3:1.2.2.3::installed
dev-haskell/MissingH-1.1.0.3:1.1.0.3::installed
dev-haskell/QuickCheck-2.4.0.1:2.4.0.1::installed
dev-haskell/array-0.3.0.2:0.3.0.2::installed
dev-haskell/bytestring-0.9.1.7:0.9.1.7::installed
dev-haskell/containers-0.4.0.0:0.4.0.0::installed
dev-haskell/extensible-exceptions-0.1.1.2:0.1.1.2::installed
dev-haskell/filepath-1.2.0.0:1.2.0.0::installed
dev-haskell/hslogger-1.1.3:0::installed
dev-haskell/mtl-2.0.1.0:2.0.1.0::installed
dev-haskell/network-2.3.0.1:2.3.0.1::installed
dev-haskell/old-locale-1.0.0.2:1.0.0.2::installed
dev-haskell/parsec-3.1.0:3.1.0::installed
dev-haskell/pcre-light-0.4:0::installed
dev-haskell/regex-base-0.93.2:0.93.2::installed
dev-haskell/regex-compat-0.93.1:0.93.1::installed
dev-haskell/regex-posix-0.94.4:0.94.4::installed
dev-haskell/syb-0.3:0.3::installed
dev-haskell/transformers-0.2.2.0:0.2.2.0::installed
dev-haskell/utf8-string-0.3.6:0.3.6::installed

(some stuff listed here as ::installed, but contains no files, since these packages detect whether ghc-7.0.2 already comes with the same/newer package version)

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Sun Apr 3 06:57:02 2011
Given that the softlinks contain all needed information (if the object exists, locally), an emergency way to get files "out" of git-annex would be nice. I am aware that one can script it, but a canonical way is always better, especially when things go south.
Sounds like you probably didn't commit after the fsck, or didn't push so the other repository did not know the first had the content again -- but I'm not 100% sure.
Comment by http://joey.kitenet.net/ Thu May 12 01:01:34 2011
make clean and rebuild worked... Thank you
joey@kitenet.net (hope I can make sense of dtruss output)
Comment by http://joey.kitenet.net/ Wed Feb 9 19:47:30 2011
@gernot step 0 is to upgrade git-annex to current git, on all systems where you use it, in case that wasn't clear.
Comment by http://joey.kitenet.net/ Sun Apr 3 16:53:51 2011

I meant to say in it wasn't reliable when I was following the instructions for "Comment 12". I did find that just doing a "git annex copy -t externalusb ." then a "git annex drop ." from the root of my cloned and "none trusted" annexed repos to be more reliable, it just means I temporarily need a load of space to get myself out of my earlier mess.

On testing this bug fix, I found a minor behavioural issue with git annex copy -f REMOTE . doesn't work as expected

I also failed to mention, that in the case when i have stray log files after what has happened in comment 2, I get this left over after a commit when git is confused...

jtang@x00:~/sources $ git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#   modified:   .git-annex/1G/X3/WORM-s309910751-m1301311322--l_fcompxe_ia32_2011.2.137.tgz.log
#   modified:   .git-annex/3W/Xf/WORM-s805764902-m1301312756--l_cproc_p_11.1.075_intel64.log
#   modified:   .git-annex/9Q/Wz/WORM-s1234430253-m1301311891--l_ccompxe_2011.2.137.log
#   modified:   .git-annex/FQ/4z/WORM-s318168323-m1301310848--l_cprof_p_11.1.075_ia64.log
#   modified:   .git-annex/FV/0P/WORM-s710135470-m1301311835--l_ccompxe_intel64_2011.2.137.log
#   modified:   .git-annex/Jk/zK/WORM-s374617670-m1301312705--l_ipp_7.0.2.137_intel64.log
#   modified:   .git-annex/Jx/qM/WORM-s599386592-m1301310731--l_fcompxe_2011.2.137.tgz.log
#   modified:   .git-annex/KX/w1/WORM-s35976002-m1301312193--l_tbb_3.0.6.174.log
#   modified:   .git-annex/VK/kv/WORM-s584342291-m1301312669--l_cproc_p_11.1.075_ia64.log
#   modified:   .git-annex/Vw/jK/WORM-s15795178-m1301310913--w_flm_p_1.0.011_intel64.zip.log
#   modified:   .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log
#   modified:   .git-annex/vW/v1/WORM-s736986678-m1301312794--l_cproc_p_11.1.075_ia32.log
#
no changes added to commit (use "git add" and/or "git commit -a")

Up until now I have just been updating the status of the staged files by hand and commiting it on my mac x00, this probably isn't helping. I'd rather not lose the tracking information.

Currently fsck silently ignores --to/--from. It should at least complain if it is not supported.

Comment by http://ertai.myopenid.com/ Sat Jun 25 16:20:44 2011

Thanks to your feedback, I got it going.

Maybe those two should be added to the 'OSX how-to' in the forum

[realizes pcre-light is needed but pcre not installed on my mac]
sudo port install pcre
sudo cabal install pcre-light

[tests are failing, need haskell's quickcheck]
sudo cabal install quickcheck

I think I know how I got myself into this mess... I was on my mac workstation and I had just pulled in a change set from another repo on a linux workstation after I had a made a bunch of moves. here's a bit of a log of what happened...

jtang@x00:~/sources $ git pull cports-devel master
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: No xauth data; using fake authentication data for X11 forwarding.
remote: Counting objects: 4195, done.
remote: Compressing objects: 100% (1135/1135), done.
remote: Total 2582 (delta 866), reused 2576 (delta 860)
Receiving objects: 100% (2582/2582), 229.42 KiB | 111 KiB/s, done.
Resolving deltas: 100% (866/866), completed with 9 local objects.
From cports-devel:/home/people/jtang/sources
 * branch            master     -> FETCH_HEAD
Updating 319df99..ab0a98c
error: Your local changes to the following files would be overwritten by merge:
    .git-annex/09/5X/WORM-s361516678-m1301310614--l_fcompxe_intel64_2011.2.137.tgz.log
    .git-annex/43/2g/WORM-s19509673-m1301310496--l_fcompxe_2011.2.137_redist.tgz.log
    .git-annex/4J/qF/WORM-s18891115-m1301310934--w_flm_p_1.0.011_ia64.zip.log
    .git-annex/87/w1/WORM-s12212473-m1301310909--w_flm_p_1.0.011_ia32.zip.log
    .git-annex/99/Jq/WORM-s194345957-m1301310926--l_mkl_10.3.2.137_ia32.log
    .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log
    .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log
    .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log
    .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log
    .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log
    .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log
    .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log
    .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log
    .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log
    .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log
    .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log
Please, commit your changes or stash them before you can merge.
error: Your local changes to the following files would be overwritten by merge:
    .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log
    .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log
    .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log
    .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log
    .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log
    .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log
    .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log
    .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log
    .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log
    .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log
    .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log
    .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log
    .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log
    .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log
    .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log
    .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log
    .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log
    .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log
    .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log
    .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log
    .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log
    .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log
    .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log
    .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log
    .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log
    .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log
    .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log
    .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log
    .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log
    .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log
Please, commit your changes or stash them before you can merge.
error: The following untracked working tree files would be overwritten by merge:
    .git-annex/1g/X3/WORM-s309910751-m1301311322--l_fcompxe_ia32_2011.2.137.tgz.log
    .git-annex/3w/Xf/WORM-s805764902-m1301312756--l_cproc_p_11.1.075_intel64.log
    .git-annex/9Q/Wz/WORM-s1234430253-m1301311891--l_ccompxe_2011.2.137.log
    .git-annex/FQ/4z/WORM-s318168323-m1301310848--l_cprof_p_11.1.075_ia64.log
    .git-annex/FV/0P/WORM-s710135470-m1301311835--l_ccompxe_intel64_2011.2.137.log
    .git-annex/Jx/qM/WORM-s599386592-m1301310731--l_fcompxe_2011.2.137.tgz.log
    .git-annex/KX/w1/WORM-s35976002-m1301312193--l_tbb_3.0.6.174.log
    .git-annex/Vw/jK/WORM-s15795178-m1301310913--w_flm_p_1.0.011_intel64.zip.log
    .git-annex/jK/zK/WORM-s374617670-m1301312705--l_ipp_7.0.2.137_intel64.log
    .git-annex/vK/kv/WORM-s584342291-m1301312669--l_cproc_p_11.1.075_ia64.log
    .git-annex/vw/v1/WORM-s736986678-m1301312794--l_cproc_p_11.1.075_ia32.log
    .git-annex/zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log
Please move or remove them before you can merge.
Aborting
1|jtang@x00:~/sources $ git status
# On branch master
# Your branch is ahead of 'origin/master' by 2 commits.
#
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#   modified:   .git-annex/09/5X/WORM-s361516678-m1301310614--l_fcompxe_intel64_2011.2.137.tgz.log
#   modified:   .git-annex/43/2g/WORM-s19509673-m1301310496--l_fcompxe_2011.2.137_redist.tgz.log
#   modified:   .git-annex/4J/qF/WORM-s18891115-m1301310934--w_flm_p_1.0.011_ia64.zip.log
#   modified:   .git-annex/87/w1/WORM-s12212473-m1301310909--w_flm_p_1.0.011_ia32.zip.log
#   modified:   .git-annex/99/Jq/WORM-s194345957-m1301310926--l_mkl_10.3.2.137_ia32.log
#   modified:   .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log
#   modified:   .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log
#   modified:   .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log
#   modified:   .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log
#   modified:   .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log
#   modified:   .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log
#   modified:   .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log
#   modified:   .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log
#   modified:   .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log
#   modified:   .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log
#   modified:   .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log
#
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#   modified:   .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log
#   modified:   .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log
#   modified:   .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log
#   modified:   .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log
#   modified:   .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log
#   modified:   .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log
#   modified:   .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log
#   modified:   .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log
#   modified:   .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log
#   modified:   .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log
#   modified:   .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log
#   modified:   .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log
#   modified:   .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log
#   modified:   .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log
#   modified:   .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log
#   modified:   .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log
#   modified:   .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log
#   modified:   .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log
#   modified:   .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log
#   modified:   .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log
#   modified:   .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log
#   modified:   .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log
#   modified:   .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log
#   modified:   .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log
#   modified:   .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log
#   modified:   .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log
#   modified:   .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log
#   modified:   .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log
#   modified:   .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log
#   modified:   .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log
#   modified:   .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#   .git-annex/1G/X3/
#   .git-annex/3W/Xf/
#   .git-annex/9q/Wz/
#   .git-annex/Fq/4z/
#   .git-annex/Jk/zK/
#   .git-annex/Kx/w1/
#   .git-annex/VK/kv/
#   .git-annex/fv/0P/
#   .git-annex/jX/qM/
#   .git-annex/vW/jK/
#   .git-annex/vW/v1/
jtang@x00:~/sources $ git commit -a -m "snap"
[master 45f254a] snap
 47 files changed, 64 insertions(+), 30 deletions(-)
jtang@x00:~/sources $ git status
# On branch master
# Your branch is ahead of 'origin/master' by 3 commits.
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#   .git-annex/1G/X3/
#   .git-annex/3W/Xf/
#   .git-annex/9q/Wz/
#   .git-annex/Fq/4z/
#   .git-annex/Jk/zK/
#   .git-annex/Kx/w1/
#   .git-annex/VK/kv/
#   .git-annex/fv/0P/
#   .git-annex/jX/qM/
#   .git-annex/vW/jK/
#   .git-annex/vW/v1/
nothing added to commit but untracked files present (use "git add" to track)
jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://fraggod.pip.verisignlabs.com.pip.verisignlabs.com/ Thu Apr 7 13:44:36 2011
Finally got a chance to try to reproduce this. I followed your recipe exactly in a clean squeeze chroot. monadIO was not installed, but git-annex built ok, using monad-control.
Comment by http://joey.kitenet.net/ Wed Aug 17 04:56:30 2011

S3 doesn't support encryption at all, yet.

It certainly makes sense to use a different portion of the encrypted secret key for HMAC than is uses as the gpg symmetric encryption key.

The two keys used in HMAC would be the secret key and the key/value key for the content bexás..2.137_ia32.log # modified: .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log # modified: .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log # modified: .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log # modified: .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log # modified: .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log # modified: .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log # modified: .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log # modified: .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log # modified: .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log # modified: .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log # modified: .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log # modified: .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log # modified: .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log # modified: .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log # modified: .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log # modified: .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log # modified: .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log # modified: .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log # modified: .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log # modified: .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log # modified: .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log # modified: .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log # modified: .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log # modified: .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log # modified: .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log # modified: .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log # modified: .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log # modified: .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log # modified: .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log # modified: .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log # modified: .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log # modified: .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log # modified: .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log # modified: .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log # modified: .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log # modified: .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log # modified: .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log # modified: .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log # modified: .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log # modified: .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log # modified: .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ jtang@x00:~/sources $ git commit -a -m "snap" [master 45f254a] snap 47 files changed, 64 insertions(+), 30 deletions(-) jtang@x00:~/sources $ git status # On branch master # Your branch is ahead of 'origin/master' by 3 commits. # # Untracked files: # (use "git add ..." to include in what will be committed) # # .git-annex/1G/X3/ # .git-annex/3W/Xf/ # .git-annex/9q/Wz/ # .git-annex/Fq/4z/ # .git-annex/Jk/zK/ # .git-annex/Kx/w1/ # .git-annex/VK/kv/ # .git-annex/fv/0P/ # .git-annex/jX/qM/ # .git-annex/vW/jK/ # .git-annex/vW/v1/ nothing added to commit but untracked files present (use "git add" to track) jtang@x00:~/sources $ git pull

Despite status listing S3 support, your git-annex is actually built with S3stub, probably because it failed to find the necessary S3 module at build time. Rebuild git-annex and watch closely, you'll see "** building without S3 support". Look above that for the error and fix it.

It was certianly a bug that it showed S3 as supported when built without it. I've fixed that.

Comment by http://joey.kitenet.net/ Fri Jan 6 03:04:35 2012

If you try to clone a git repo that has a symlink over to a VFAT filesystem, you get (in its place) a regular file that contains the name of the symlink target. So why can't git-annex use that? I could still do git annex get on this file, git annex would still "know" that it's a symlink, and could replace it with a copy of the real file (instead of putting it in .git/annex).

I know if it were that simple, someone would have done it already, so what am I missing? I guess trying to get the file FROM the repository would fail because it wouldn't find the file in .git/annex? Couldn't you store a reverse mapping? You wouldn't be able to move the file around, but you already lose that once you give up symlinks. It would also be a little harder to tell which symlinks were "dangling"; I don't see an easy way to get around that. It would still be better than a bare repo..

Comment by http://ethan.betacantrips.com/ Wed Jun 8 20:59:38 2011
When "make test" fails to run any tests at all, it should not return exit code 0. This behavior is quite misleading, and it means that automated build systems are not going to detect the fact that the test suit could not be run.
Comment by http://peter-simons.myopenid.com/ Tue Jan 3 18:09:38 2012
I don't know what these problems forking could be. Can you strace it?
Comment by http://joey.kitenet.net/ Wed Feb 9 15:04:50 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Yes, I've moved it to OSX page where anyone can update it in this wiki, and added your improvements.
Comment by http://joey.kitenet.net/ Sun Feb 6 17:39:52 2011
Just tried it out on my mac and it's working again. I guess this issue could be closed for now.
After another run, i am at 8909 files in the remote, now.
Picking up the automagic encryption idea for annex remotes, this would allow you to host a branchable-esque git-annex hosting service. (Nexenta with ZFS is a cheap and reliable option until btrfs becomes stable in a year or five).

I just noticed this issue, and was wondering what the current status is.

% ls -l 04\ -\ Orixás.mp3
-rw-r--r-- 1 adam users 8377816 Jul 12  2007 04 - Orixás.mp3
% echo 04\ -\ Orixás.mp3 | od -c
0000000   0   4       -       O   r   i   x 303 241   s   .   m   p   3
0000020  \n
0000021
% git annex add 04\ -\ Orixás.mp3
git-annex: /home/adam/music/RotC/transcribe/04 - Orixás.mp3: getSymbolicLinkStatus: does not exist (No such file or directory)
Comment by http://adamspiers.myopenid.com/ Sat Dec 24 01:05:07 2011

Finally got around to report the issue to GHC tracker.

Looks quite alike (at least to the haskell-illiterate person like me) to a highest-priority issue that's hanging right at the top of the list. There are other similar reports, but they seem to be either related to PowerPC Macs, closed as invalid or due to needinfo inactivity.

Guess any further discussion belongs there, unless ghc developers will bounce it back. Thanks a lot for your help, Joey, and for sharing a great thing that git-annex is.

Comment by http://frag