copy doesn't scale

It seems that git-annex copies every individual file in a separate transaction. This is quite costly for mass transfers: each file involves a separate rsync invocation and the creation of a new commit. Even with a meager thousand files or so in the annex, I have to wait for fifteen minutes to copy the contents to another disk, simply because every individual file involves some disk thrashing. Also, it seems suspicious that the git-annex branch would get a thousands commits of history from the simple procedure of copying everything to a new repository. Surely it would be better to first copy everything and then create only a single commit that registers the changes to the files' availability?

git-annex is very careful to commit as infrequently as possible, and the current version makes 1 commit after all the copies are complete, even if it transferred a billion files. The only overhead incurred for each file is writing a journal file. You must have an old version. --Joey

(I'm also not quite clear on why rsync is being used when both repositories are local. It seems to be just overhead.)

Even when copying to another disk it's often on some slow bus, and the file is by definition large. So it's nice to support resumes of interrupted transfers of files. Also because rsync has a handy progress display that is hard to get with cp.

(However, if the copy is to another directory in the same disk, it does use cp, and even supports really fast copies on COW filesystems.) --Joey

Oneshot mode is now implemented, making git-annex-shell and other short lifetime processes not bother with committing changes. done --Joey

Update: Now it makes one commit at the very end of such a mass transfer. --Joey

comment 1

To me it very much seems that a commit per file is indeed created at the remote end, although not at the local end. See the following transcript: https://gist.github.com/1691714.

Comment by https://www.google.com/accounts/o8/id?id=AItOawk6QAwUsFHpr3Km1yQbg8hf3S7RDYf7hX4 — Sat Jan 28 00:17:37 2012

comment 2

Ah, I see, I was not thinking about the location log update that's done on the remote side.

For transfers over ssh, that's a separate git-annex-shell invoked per change. For local-local transfers, it's all done in a single process but it spins up a state to handle the remote and then immediately shuts it down, also gen././@LongLink0000000000000000000000000000020500000000000011562 Lustar rootroot./usr/share/doc/git-annex/html/bugs/fix_for_makefile_to_check_if_OS_is_linux_or_not___40__relates_to_the_new_inotify_flag__41__.html./usr/share/doc/git-annex/html/bugs/fix_for_makefile_to_check_if_OS_is_linux_or_not___40__relates_to0000644000000000000000000000450111773543307033006 0ustar rootroot fix for makefile to check if OS is linux or not (relates to the new inotify flag)

Since the watch branch is now merged into master, it doesn't quite build cleanly on non-linux systems anymore. So here's a small change to the Makefile to work around the problem for now.

From 707cb47744775c324060febe11987db5f10ed9ff Mon Sep 17 00:00:00 2001
From: Jimmy Tang 
Date: Mon, 18 Jun 2012 09: