git-fast-import(1)
==================

NAME
----
git-fast-import - Backend for fast Git data importers


SYNOPSIS
--------
frontend | 'git-fast-import' [options]

DESCRIPTION
-----------
This program is usually not what the end user wants to run directly.
Most end users want to use one of the existing frontend programs,
which parses a specific type of foreign source and feeds the contents
stored there to git-fast-import.

fast-import reads a mixed command/data stream from standard input and
writes one or more packfiles directly into the current repository.
When EOF is received on standard input, fast import writes out
updated branch and tag refs, fully updating the current repository
with the newly imported data.

The fast-import backend itself can import into an empty repository (one that
has already been initialized by linkgit:git-init[1]) or incrementally
update an existing populated repository.  Whether or not incremental
imports are supported from a particular foreign source depends on
the frontend program in use.


OPTIONS
-------
--date-format=<fmt>::
	Specify the type of dates the frontend will supply to
	fast-import within `author`, `committer` and `tagger` commands.
	See ``Date Formats'' below for details about which formats
	are supported, and their syntax.

--force::
	Force updating modified existing branches, even if doing
	so would cause commits to be lost (as the new commit does
	not contain the old commit).

--max-pack-size=<n>::
	Maximum size of each output packfile, expressed in MiB.
	The default is 4096 (4 GiB) as that is the maximum allowed
	packfile size (due to file format limitations). Some
	importers may wish to lower this, such as to ensure the
	resulting packfiles fit on CDs.

--depth=<n>::
	Maximum delta depth, for blob and tree deltification.
	Default is 10.

--active-branches=<n>::
	Maximum number of branches to maintain active at once.
	See ``Memory Utilization'' below for details.  Default is 5.

--export-marks=<file>::
	Dumps the internal marks table to <file> when complete.
	Marks are written one per line as `:markid SHA-1`.
	Frontends can use this file to validate imports after they
	have been completed, or to save the marks table across
	incremental runs.  As <file> is only opened and truncated
	at checkpoint (or completion) the same path can also be
	safely given to \--import-marks.

--import-marks=<file>::
	Before processing any input, load the marks specified in
	<file>.  The input file must exist, must be readable, and
	must use the same format as produced by \--export-marks.
	Multiple options may be supplied to import t>checkpoint</tt></h3>
<p>Forces fast-import to close the current packfile, start a new one, and to
save out all current branch refs, tags and marks.</p>
<div class="literalblock">
<div class="content">
<pre><tt>        'checkpoint' LF
        LF?</tt></pre>
</div></div>
<p>Note that fast-import automatically switches packfiles when the current
packfile reaches --max-pack-size, or 4 GiB, whichever limit is
smaller.  During an automatic packfile switch fast-import does not update
the branch refs, tags or marks.</p>
<p>As a <tt>checkpoint</tt> can require a significant amount of CPU time and
disk IO (to compute the overall pack SHA-1 checksum, generate the
corresponding index file, and update the refs) it can easily take
several minutes for a single <tt>checkpoint</tt> command to complete.</p>
<p>Frontends may choose to issue checkpoints during extremely large
and long running imports, or when they need to allow another Git
process access to a branch.  However given that a 30 GiB Subversion
repository can be loaded into Git through fast-import in about 3 hours,
explicit checkpointing may not be necessary.</p>
<p>The <tt>LF</tt> after the command is optional (it used to be required).</p>
<h3><tt>progress</tt></h3>
<p>Causes fast-import to print the entire <tt>progress</tt> line unmodified to
its standard output channel (file descriptor 1) when the command is
processed from the input stream.  The command otherwise has no impact
on the current import, or on any of fast-import's internal state.</p>
<div class="literalblock">
<div class="content">
<pre><tt>        'progress' SP &lt;any&gt; LF
        LF?</tt></pre>
</div></div>
<p>The <tt>&lt;any&gt;</tt> part of the command may contain any sequence of bytes
that does not contain <tt>LF</tt>.  The <tt>LF</tt> after the command is optional.
Callers may wish to process the output through a tool such as sed to
remove the leading part of the line, for example:</p>
<div class="exampleblock">
<div class="exampleblock-content">
<div class="literalblock">
<div class="content">
<pre><tt>frontend | git-fast-import | sed 's/^progress //'</tt></pre>
</div></div>
</div></div>
<p>Placing a <tt>progress</tt> command immediately after a <tt>checkpoint</tt> will
inform the reader when the <tt>checkpoint</tt> has been completed and it
can safely access the refs that fast-import updated.</p>
</div>
<h2>Tips and Tricks</h2>
<div class="sectionbody">
<p>The following tips and tricks have been collected from various
users of fast-import, and are offered here as suggestions.</p>
<h3>Use One Mark Per Commit</h3>
<p>When doing a repository conversion, use a unique mark per commit
(<tt>mark :&lt;n&gt;</tt>) and supply the --export-marks option on the command
line.  fast-import will dump a file which lists every mark and the Git
object SHA-1 that corresponds to it.  If the frontend can tie
the marks back to the source repository, it is easy to verify the
accuracy and completeness of the import by comparing each Git
commit to the corresponding source revision.</p>
<p>Coming from a system such as Perforce or Subversion this should be
quite simple, as the fast-import mark can also be the Perforce changeset
number or the Subversion revision number.</p>
<h3>Freely Skip Around Branches</h3>
<p>Don't bother trying to optimize the frontend to stick to one branch
at a time during an import.  Although doing so might be slightly
faster for fast-import, it tends to increase the complexity of the frontend
code considerably.</p>
<p>The branch LRU builtin to fast-import tends to behave very well, and the
cost of activating an inactive branch is so low that bouncing around
between branches has virtually no impact on import performance.</p>
<h3>Handling Renames</h3>
<p>When importing a renamed file or directory, simply delete the old
name(s) and modify the new name(s) during the corresponding commit.
Git performs rename detection after-the-fact, rather than explicitly
during a commit.</p>
<h3>Use Tag Fixup Branches</h3>
<p>Some other SCM systems let the user create a tag from multiple
files which are not from the same commit/changeset.  Or to create
tags which are a subset of the files available in the repository.</p>
<p>Importing these tags as-is in Git is impossible without making at
least one commit which &#8220;fixes up&#8221; the files to match the content
of the tag.  Use fast-import's <tt>reset</tt> command to reset a dummy branch
outside of your normal branch space to the base commit for the tag,
then commit one or more file fixup commits, and finally tag the
dummy branch.</p>
<p>For example since all normal branches are stored under <tt>refs/heads/</tt>
name the tag fixup branch <tt>TAG_FIXUP</tt>.  This way it is impossible for
the fixup branch used by the importer to have namespace conflicts
with real branches imported from the source (the name <tt>TAG_FIXUP</tt>
is not <tt>refs/heads/TAG_FIXUP</tt>).</p>
<p>When committing fixups, consider using <tt>merge</tt> to connect the
commit(s) which are supplying file revisions to the fixup branch.
Doing so will allow tools such as <a href="git-blame.html">git-blame(1)</a> to track
through the real commit history and properly annotate the source
files.</p>
<p>After fast-import terminates the frontend will need to do <tt>rm .git/TAG_FIXUP</tt>
to remove the dummy branch.</p>
<h3>Import Now, Repack Later</h3>
<p>As soon as fast-import completes the Git repository is completely valid
and ready for use.  Typically this takes only a very short time,
even for considerably large projects (100,000+ commits).</p>
<p>However repacking the repository is necessary to improve data
locality and access performance.  It can also take hours on extremely
large projects (especially if -f and a large --window parameter is
used).  Since repacking is safe to run alongside readers and writers,
run the repack in the background and let it finish when it finishes.
There is no reason to wait to explore your new Git project!</p>
<p>If you choose to wait for the repack, don't try to run benchmarks
or performance tests until repacking is completed.  fast-import outputs
suboptimal packfiles that are simply never seen in real use
situations.</p>
<h3>Repacking Historical Data</h3>
<p>If you are repacking very old imported data (e.g. older than the
last year), consider expending some extra CPU time and supplying
--window=50 (or higher) when you run <a href="git-repack.html">git-repack(1)</a>.
This will take longer, but will also produce a smaller packfile.
You only need to expend the effort once, and everyone using your
project will benefit from the smaller repository.</p>
<h3>Include Some Progress Messages</h3>
<p>Every once in a while have your frontend emit a <tt>progress</tt> message
to fast-import.  The contents of the messages are entirely free-form,
so one suggestion would be to output the current month and year
each time the current commit date moves into the next month.
Your users will feel better knowing how much of the data stream
has been processed.</p>
</div>
<h2>Packfile Optimization</h2>
<div class="sectionbody">
<p>When packing a blob fast-import always attempts to deltify against the last
blob written.  Unless specifically arranged for by the frontend,
this will probably not be a prior version of the same file, so the
generated delta will not be the smallest possible.  The resulting
packfile will be compressed, but will not be optimal.</p>
<p>Frontends which have efficient access to all revisions of a
single file (for example reading an RCS/CVS ,v file) can choose
to supply all revisions of that file as a sequence of consecutive
<tt>blob</tt> commands.  This allows fast-import to deltify the different file
revisions against each other, saving space in the final packfile.
Marks can be used to later identify individual file revisions during
a sequence of <tt>commit</tt> commands.</p>
<p>The packfile(s) created by fast-import do not encourage good disk access
patterns.  This is caused by fast-import writing the data in the order
it is received on standard input, while Git typically organizes
data within packfiles to make the most recent (current tip) data
appear before historical data.  Git also clusters commits together,
speeding up revision traversal through better cache locality.</p>
<p>For this reason it is strongly recommended that users repack the
repository with <tt>git repack -a -d</tt> after fast-import completes, allowing
Git to reorganize the packfiles for faster data access.  If blob
deltas are suboptimal (see above) then also adding the <tt>-f</tt> option
to force recomputation of all deltas can significantly reduce the
final packfile size (30-50% smaller can be quite typical).</p>
</div>
<h2>Memory Utilization</h2>
<div class="sectionbody">
<p>There are a number of factors which affect how much memory fast-import
requires to perform an import.  Like critical sections of core
Git, fast-import uses its own memory allocators to amortize any overheads
associated with malloc.  In practice fast-import tends to amortize any
malloc overheads to 0, due to its use of large block allocations.</p>
<h3>per object</h3>
<p>fast-import maintains an in-memory structure for every object written in
this execution.  On a 32 bit system the structure is 32 bytes,
on a 64 bit system the structure is 40 bytes (due to the larger
pointer sizes).  Objects in the table are not deallocated until
fast-import terminates.  Importing 2 million objects on a 32 bit system
will require approximately 64 MiB of memory.</p>
<p>The object table is actually a hashtable keyed on the object name
(the unique SHA-1).  This storage configuration allows fast-import to reuse
an existing or already written object and avoid writing duplicates
to the output packfile.  Duplicate blobs are surprisingly common
in an import, typically due to branch merges in the source.</p>
<h3>per mark</h3>
<p>Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
bytes, depending on pointer size) per mark.  Although the array
is sparse, frontends are still strongly encouraged to use marks
between 1 and n, where n is the total number of marks required for
this import.</p>
<h3>per branch</h3>
<p>Branches are classified as active and inactive.  The memory usage
of the two classes is significantly different.</p>
<p>Inactive branches are stored in a structure which uses 96 or 120
bytes (32 bit or 64 bit systems, respectively), plus the length of
the branch name (typically under 200 bytes), per branch.  fast-import will
easily handle as many as 10,000 inactive branches in under 2 MiB
of memory.</p>
<p>Active branches have the same overhead as inactive branches, but
also contain copies of every tree that has been recently modified on
that branch.  If subtree <tt>include</tt> has not been modified since the
branch became active, its contents will not be loaded into memory,
but if subtree <tt>src</tt> has been modified by a commit since the branch
became active, then its contents will be loaded in memory.</p>
<p>As active branches store metadata about the files contained on that
branch, their in-memory storage size can grow to a considerable size
(see below).</p>
<p>fast-import automatically moves active branches to inactive status based on
a simple least-recently-used algorithm.  The LRU chain is updated on
each <tt>commit</tt> command.  The maximum number of active branches can be
increased or decreased on the command line with --active-branches=.</p>
<h3>per active tree</h3>
<p>Trees (aka directories) use just 12 bytes of memory on top of the
memory required for their entries (see &#8220;per active file&#8221; below).
The cost of a tree is virtually 0, as its overhead amortizes out
over the individual file entries.</p>
<h3>per active file entry</h3>
<p>Files (and pointers to subtrees) within active trees require 52 or 64
bytes (32/64 bit platforms) per entry.  To conserve space, file and
tree names are pooled in a common string table, allowing the filename
&#8220;Makefile&#8221; to use just 16 bytes (after including the string header
overhead) no matter how many times it occurs within the project.</p>
<p>The active branch LRU, when coupled with the filename string pool
and lazy loading of subtrees, allows fast-import to efficiently import
projects with 2,000+ branches and 45,114+ files in a very limited
memory footprint (less than 2.7 MiB per active branch).</p>
</div>
<h2>Author</h2>
<div class="sectionbody">
<p>Written by Shawn O. Pearce &lt;spearce@spearce.org&gt;.</p>
</div>
<h2>Documentation</h2>
<div class="sectionbody">
<p>Documentation by Shawn O. Pearce &lt;spearce@spearce.org&gt;.</p>
</div>
<h2>GIT</h2>
<div class="sectionbody">
<p>Part of the <a href="git.html">git(7)</a> suite</p>
</div>
<div id="footer">
<div id="footer-text">
Last updated 19-Feb-2008 15:56:20 UTC
</div>
</div>
</body>
</html>
                                                                                                                                                                                                     ./usr/share/doc/git-doc/git-fast-import.txt                                                         0000644 0000000 0000000 00000120440 10756577465 020775  0                                                                                                    ustar   root                            root                            0000000 0000000                                                                                                                                                                        git-fast-import(1)
==================

NAME
----
git-fast-import - Backend for fast Git data importers


SYNOPSIS
--------
frontend | 'git-fast-import' [options]

DESCRIPTION
-----------
This program is usually not what the end user wants to run directly.
Most end users want to use one of the existing frontend programs,
which parses a specific type of foreign source and feeds the contents
stored there to git-fast-import.

fast-import reads a mixed command/data stream from standard input and
writes one or more packfiles directly into the current repository.
When EOF is received on standard input, fast import writes out
updated branch and tag refs, fully updating the current repository
with the newly imported data.

The fast-import backend itself can import into an empty repository (one that
has already been initialized by linkgit:git-init[1]) or incrementally
update an existing populated repository.  Whether or not incremental
imports are supported from a particular foreign source depends on
the frontend program in use.


OPTIONS
-------
--date-format=<fmt>::
	Specify the type of dates the frontend will supply to
	fast-import within `author`, `committer` and `tagger` commands.
	See ``Date Formats'' below for details about which formats
	are supported, and their syntax.

--force::
	Force updating modified existing branches, even if doing
	so would cause commits to be lost (as the new commit does
	not contain the old commit).

--max-pack-size=<n>::
	Maximum size of each output packfile, expressed in MiB.
	The default is 4096 (4 GiB) as that is the maximum allowed
	packfile size (due to file format limitations). Some
	importers may wish to lower this, such as to ensure the
	resulting packfiles fit on CDs.

--depth=<n>::
	Maximum delta depth, for blob and tree deltification.
	Default is 10.

--active-branches=<n>::
	Maximum number of branches to maintain active at once.
	See ``Memory Utilization'' below for details.  Default is 5.

--export-marks=<file>::
	Dumps the internal marks table to <file> when complete.
	Marks are written one per line as `:markid SHA-1`.
	Frontends can use this file to validate imports after they
	have been completed, or to save the marks table across
	incremental runs.  As <file> is only opened and truncated
	at checkpoint (or completion) the same path can also be
	safely given to \--import-marks.

--import-marks=<file>::
	Before processing any input, load the marks specified in
	<file>.  The input file must exist, must be readable, and
	must use the same format as produced by \--export-marks.
	Multiple options may be supplied to import t>checkpoint</tt></h3>
<p>Forces fast-import to close the current packfile, start a new one, and to
save out all current branch refs, tags and marks.</p>
<div class="literalblock">
<div class="content">
<pre><tt>        'checkpoint' LF
        LF?</tt></pre>
</div></div>
<p>Note that fast-import automatically switches packfiles when the current
packfile reaches --max-pack-size, or 4 GiB, whichever limit is
smaller.  During an automatic packfile switch fast-import does not update
the branch refs, tags or marks.</p>
<p>As a <tt>checkpoint</tt> can require a significant amount of CPU time and
disk IO (to compute the overall pack SHA-1 checksum, generate the
corresponding index file, and update the refs) it can easily take
several minutes for a single <tt>checkpoint</tt> command to complete.</p>
<p>Frontends may choose to issue checkpoints during extremely large
and long running imports, or when they need to allow another Git
process access to a branch.  However given that a 30 GiB Subversion
repository can be loaded into Git through fast-import in about 3 hours,
explicit checkpointing may not be necessary.</p>
<p>The <tt>LF</tt> after the command is optional (it used to be required).</p>
<h3><tt>progress</tt></h3>
<p>Causes fast-import to print the entire <tt>progress</tt> line unmodified to
its standard output channel (file descriptor 1) when the command is
processed from the input stream.  The command otherwise has no impact
on the current import, or on any of fast-import's internal state.</p>
<div class="literalblock">
<div class="content">
<pre><tt>        'progress' SP &lt;any&gt; LF
        LF?</tt></pre>
</div></div>
<p>The <tt>&lt;any&gt;</tt> part of the command may contain any sequence of bytes
that does not contain <tt>LF</tt>.  The <tt>LF</tt> after the command is optional.
Callers may wish to process the output through a tool such as sed to
remove the leading part of the line, for example:</p>
<div class="exampleblock">
<div class="exampleblock-content">
<div class="literalblock">
<div class="content">
<pre><tt>frontend | git-fast-import | sed 's/^progress //'</tt></pre>
</div></div>
</div></div>
<p>Placing a <tt>progress</tt> command immediately after a <tt>checkpoint</tt> will
inform the reader when the <tt>checkpoint</tt> has been completed and it
can safely access the refs that fast-import updated.</p>
</div>
<h2>Tips and Tricks</h2>
<div class="sectionbody">
<p>The following tips and tricks have been collected from various
users of fast-import, and are offered here as suggestions.</p>
<h3>Use One Mark Per Commit</h3>
<p>When doing a repository conversion, use a unique mark per commit
(<tt>mark :&lt;n&gt;</tt>) and supply the --export-marks option on the command
line.  fast-import will dump a file which lists every mark and the Git
object SHA-1 that corresponds to it.  If the frontend can tie
the marks back to the source repository, it is easy to verify the
accuracy and completeness of the import by comparing each Git
commit to the corresponding source revision.</p>
<p>Coming from a system such as Perforce or Subversion this should be
quite simple, as the fast-import mark can also be the Perforce changeset
number or the Subversion revision number.</p>
<h3>Freely Skip Around Branches</h3>
<p>Don't bother trying to optimize the frontend to stick to one branch
at a time during an import.  Although doing so might be slightly
faster for fast-import, it tends to increase the complexity of the frontend
code considerably.</p>
<p>The branch LRU builtin to fast-import tends to behave very well, and the
cost of activating an inactive branch is so low that bouncing around
between branches has virtually no impact on import performance.</p>
<h3>Handling Renames</h3>
<p>When importing a renamed file or directory, simply delete the old
name(s) and modify the new name(s) during the corresponding commit.
Git performs rename detection after-the-fact, rather than explicitly
during a commit.</p>
<h3>Use Tag Fixup Branches</h3>
<p>Some other SCM systems let the user create a tag from multiple
files which are not from the same commit/changeset.  Or to create
tags which are a subset of the files available in the repository.</p>
<p>Importing these tags as-is in Git is impossible without making at
least one commit which &#8220;fixes up&#8221; the files to match the content
of the tag.  Use fast-import's <tt>reset</tt> command to reset a dummy branch
outside of your normal branch space to the base commit for the tag,
then commit one or more file fixup commits, and finally tag the
dummy branch.</p>
<p>For example since all normal branches are stored under <tt>refs/heads/</tt>
name the tag fixup branch <tt>TAG_FIXUP</tt>.  This way it is impossible for
the fixup branch used by the importer to have namespace conflicts
with real branches imported from the source (the name <tt>TAG_FIXUP</tt>
is not <tt>refs/heads/TAG_FIXUP</tt>).</p>
<p>When committing fixups, consider using <tt>merge</tt> to connect the
commit(s) which are supplying file revisions to the fixup branch.
Doing so will allow tools such as <a href="git-blame.html">git-blame(1)</a> to track
through the real commit history and properly annotate the source
files.</p>
<p>After fast-import terminates the frontend will need to do <tt>rm .git/TAG_FIXUP</tt>
to remove the dummy branch.</p>
<h3>Import Now, Repack Later</h3>
<p>As soon as fast-import completes the Git repository is completely valid
and ready for use.  Typically this takes only a very short time,
even for considerably large projects (100,000+ commits).</p>
<p>However repacking the repository is necessary to improve data
locality and access performance.  It can also take hours on extremely
large projects (especially if -f and a large --window parameter is
used).  Since repacking is safe to run alongside readers and writers,
run the repack in the background and let it finish when it finishes.
There is no reason to wait to explore your new Git project!</p>
<p>If you choose to wait for the repack, don't try to run benchmarks
or performance tests until repacking is completed.  fast-import outputs
suboptimal packfiles that are simply never seen in real use
situations.</p>
<h3>Repacking Historical Data</h3>
<p>If you are repacking very old imported data (e.g. older than the
last year), consider expending some extra CPU time and supplying
--window=50 (or higher) when you run <a href="git-repack.html">git-repack(1)</a>.
This will take longer, but will also produce a smaller packfile.
You only need to expend the effort once, and everyone using your
project will benefit from the smaller repository.</p>
<h3>Include Some Progress Messages</h3>
<p>Every once in a while have your frontend emit a <tt>progress</tt> message
to fast-import.  The contents of the messages are entirely free-form,
so one suggestion would be to output the current month and year
each time the current commit date moves into the next month.
Your users will feel better knowing how much of the data stream
has been processed.</p>
</div>
<h2>Packfile Optimization</h2>
<div class="sectionbody">
<p>When packing a blob fast-import always attempts to deltify against the last
blob written.  Unless specifically arranged for by the frontend,
this will probably not be a prior version of the same file, so the
generated delta will not be the smallest possible.  The resulting
packfile will be compressed, but will not be optimal.</p>
<p>Frontends which have efficient access to all revisions of a
single file (for example reading an RCS/CVS ,v file) can choose
to supply all revisions of that file as a sequence of consecutive
<tt>blob</tt> commands.  This allows fast-import to deltify the different file
revisions against each other, saving space in the final packfile.
Marks can be used to later identify individual file revisions during
a sequence of <tt>commit</tt> commands.</p>
<p>The packfile(s) created by fast-import do not encourage good disk access
patterns.  This is caused by fast-import writing the data in the order
it is received on standard input, while Git typically organizes
data within packfiles to make the most recent (current tip) data
appear before historical data.  Git also clusters commits together,
speeding up revision traversal through better cache locality.</p>
<p>For this reason it is strongly recommended that users repack the
repository with <tt>git repack -a -d</tt> after fast-import completes, allowing
Git to reorganize the packfiles for faster data access.  If blob
deltas are suboptimal (see above) then also adding the <tt>-f</tt> option
to force recomputation of all deltas can significantly reduce the
final packfile size (30-50% smaller can be quite typical).</p>
</div>
<h2>Memory Utilization</h2>
<div class="sectionbody">
<p>There are a number of factors which affect how much memory fast-import
requires to perform an import.  Like critical sections of core
Git, fast-import uses its own memory allocators to amortize any overheads
associated with malloc.  In practice fast-import tends to amortize any
malloc overheads to 0, due to its use of large block allocations.</p>
<h3>per object</h3>
<p>fast-import maintains an in-memory structure for every object written in
this execution.  On a 32 bit system the structure is 32 bytes,
on a 64 bit system the structure is 40 bytes (due to the larger
pointer sizes).  Objects in the table are not deallocated until
fast-import terminates.  Importing 2 million objects on a 32 bit system
will require approximately 64 MiB of memory.</p>
<p>The object table is actually a hashtable keyed on the object name
(the unique SHA-1).  This storage configuration allows fast-import to reuse
an existing or already written object and avoid writing duplicates
to the output packfile.  Duplicate blobs are surprisingly common
in an import, typically due to branch merges in the source.</p>
<h3>per mark</h3>
<p>Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
bytes, depending on pointer size) per mark.  Although the array
is sparse, frontends are still strongly encouraged to use marks
between 1 and n, where n is the total number of marks required for
this import.</p>
<h3>per branch</h3>
<p>Branches are classified as active and inactive.  The memory usage
of the two classes is significantly different.</p>
<p>Inactive branches are stored in a structure which uses 96 or 120
bytes (32 bit or 64 bit systems, respectively), plus the length of
the branch name (typically under 200 bytes), per branch.  fast-import will
easily handle as many as 10,000 inactive branches in under 2 MiB
of memory.</p>
<p>Active branches have the same overhead as inactive branches, but
also contain copies of every tree that has been recently modified on
that branch.  If subtree <tt>include</tt> has not been modified since the
branch became active, its contents will not be loaded into memory,
but if subtree <tt>src</tt> has been modified by a commit since the branch
became active, then its contents will be loaded in memory.</p>
<p>As active branches store metadata about the files contained on that
branch, their in-memory storage size can grow to a considerable size
(see below).</p>
<p>fast-import automatically moves active branches to inactive status based on
a simple least-recently-used algorithm.  The LRU chain is updated on
each <tt>commit</tt> command.  The maximum number of active branches can be
increased or decreased on the command line with --active-branches=.</p>
<h3>per active tree</h3>
<p>Trees (aka directories) use just 12 bytes of memory on top of the
memory required for their entries (see &#8220;per active file&#8221; below).
The cost of a tree is virtually 0, as its overhead amortizes out
over the individual file entries.</p>
<h3>per active file entry</h3>
<p>Files (and pointers to subtrees) within active trees require 52 or 64
bytes (32/64 bit platforms) per entry.  To conserve space, file and
tree names are pooled in a common string table, allowing the filename
&#8220;Makefile&#8221; to use just 16 bytes (after including the string header
overhead) no matter how many times it occurs within the project.</p>
<p>The active branch LRU, when coupled with the filename string pool
and lazy loading of subtrees, allows fast-import to efficiently import
projects with 2,000+ branches and 45,114+ files in a very limited
memory footprint (less than 2.7 MiB per active branch).</p>
</div>
<h2>Author</h2>
<div class="sectionbody">
<p>Written by Shawn O. Pearce &lt;spearce@spearce.org&gt;.</p>
</div>
<h2>Documentation</h2>
<div class="sectionbody">
<p>Documentation by Shawn O. Pearce &lt;spearce@spearce.org&gt;.</p>
</div>
<h2>GIT</h2>
<div class="sectionbody">
<p>Part of the <a href="git.html">git(7)</a> suite</p>
</div>
<div id="footer">
<div id="footer-text">
Last updated 19-Feb-2008 15:56:20 UTC
</div>
</div>
</body>
</html>
                                                                                                                                                                                                     ./usr/share/doc/git-doc/git-fast-import.txt                                                         0000644 0000000 0000000 00000120440 10756577465 020775  0                                                                                                    ustar   root                            root                            0000000 0000000                                                                                                                                                                        git-fast-import(1)
==================

NAME
----
git-fast-import - Backend for fast Git data importers


SYNOPSIS
--------
frontend | 'git-fast-import' [options]

DESCRIPTION
-----------
This program is usually not what the end user wants to run directly.
Most end users want to use one of the existing frontend programs,
which parses a specific type of foreign source and feeds the contents
stored there to git-fast-import.

fast-import reads a mixed command/data stream from standard input and
writes one or more packfiles directly into the current repository.
When EOF is received on standard input, fast import writes out
updated branch and tag refs, fully updating the current repository
with the newly imported data.

The fast-import backend itself can import into an empty repository (one that
has already been initialized by linkgit:git-init[1]) or incrementally
update an existing populated repository.  Whether or not incremental
imports are supported from a particular foreign source depends on
the frontend program in use.


OPTIONS
-------
--date-format=<fmt>::
	Specify the type of dates the frontend will supply to
	fast-import within `author`, `committer` and `tagger` commands.
	See ``Date Formats'' below for details about which formats
	are supported, and their syntax.

--force::
	Force updating modified existing branches, even if doing
	so would cause commits to be lost (as the new commit does
	not contain the old commit).

--max-pack-size=<n>::
	Maximum size of each output packfile, expressed in MiB.
	The default is 4096 (4 GiB) as that is the maximum allowed
	packfile size (due to file format limitations). Some
	importers may wish to lower this, such as to ensure the
	resulting packfiles fit on CDs.

--depth=<n>::
	Maximum delta depth, for blob and tree deltification.
	Default is 10.

--active-branches=<n>::
	Maximum number of branches to maintain active at once.
	See ``Memory Utilization'' below for details.  Default is 5.

--export-marks=<file>::
	Dumps the internal marks table to <file> when complete.
	Marks are written one per line as `:markid SHA-1`.
	Frontends can use this file to validate imports after they
	have been completed, or to save the marks table across
	incremental runs.  As <file> is only opened and truncated
	at checkpoint (or completion) the same path can also be
	safely given to \--import-marks.

--import-marks=<file>::
	Before processing any input, load the marks specified in
	<file>.  The input file must exist, must be readable, and
	must use the same format as produced by \--export-marks.
	Multiple options may be supplied to import t>checkpoint</tt></h3>
<p>Forces fast-import to close the current packfile, start a new one, and to
save out all current branch refs, tags and marks.</p>
<div class="literalblock">
<div class="content">
<pre><tt>        'checkpoint' LF
        LF?</tt></pre>
</div></div>
<p>Note that fast-import automatically switches packfiles when the current
packfile reaches --max-pack-size, or 4 GiB, whichever limit is
smaller.  During an automatic packfile switch fast-import does not update
the branch refs, tags or marks.</p>
<p>As a <tt>checkpoint</tt> can require a significant amount of CPU time and
disk IO (to compute the overall pack SHA-1 checksum, generate the
corresponding index file, and update the refs) it can easily take
several minutes for a single <tt>checkpoint</tt> command to complete.</p>
<p>Frontends may choose to issue checkpoints during extremely large
and long running imports, or when they need to allow another Git
process access to a branch.  However given that a 30 GiB Subversion
repository can be loaded into Git through fast-import in about 3 hours,
explicit checkpointing may not be necessary.</p>
<p>The <tt>LF</tt> after the command is optional (it used to be required).</p>
<h3><tt>progress</tt></h3>
<p>Causes fast-import to print the entire <tt>progress</tt> line unmodified to
its standard output channel (file descriptor 1) when the command is
processed from the input stream.  The command otherwise has no impact
on the current import, or on any of fast-import's internal state.</p>
<div class="literalblock">
<div class="content">
<pre><tt>        'progress' SP &lt;any&gt; LF
        LF?</tt></pre>
</div></div>
<p>The <tt>&lt;any&gt;</tt> part of the command may contain any sequence of bytes
that does not contain <tt>LF</tt>.  The <tt>LF</tt> after the command is optional.
Callers may wish to process the output through a tool such as sed to
remove the leading part of the line, for example:</p>
<div class="exampleblock">
<div class="exampleblock-content">
<div class="literalblock">
<div class="content">
<pre><tt>frontend | git-fast-import | sed 's/^progress //'</tt></pre>
</div></div>
</div></div>
<p>Placing a <tt>progress</tt> command immediately after a <tt>checkpoint</tt> will
inform the reader when the <tt>checkpoint</tt> has been completed and it
can safely access the refs that fast-import updated.</p>
</div>
<h2>Tips and Tricks</h2>
<div class="sectionbody">
<p>The following tips and tricks have been collected from various
users of fast-import, and are offered here as suggestions.</p>
<h3>Use One Mark Per Commit</h3>
<p>When doing a repository conversion, use a unique mark per commit
(<tt>mark :&lt;n&gt;</tt>) and supply the --export-marks option on the command
line.  fast-import will dump a file which lists every mark and the Git
object SHA-1 that corresponds to it.  If the frontend can tie
the marks back to the source repository, it is easy to verify the
accuracy and completeness of the import by comparing each Git
commit to the corresponding source revision.</p>
<p>Coming from a system such as Perforce or Subversion this should be
quite simple, as the fast-import mark can also be the Perforce changeset
number or the Subversion revision number.</p>
<h3>Freely Skip Around Branches</h3>
<p>Don't bother trying to optimize the frontend to stick to one branch
at a time during an import.  Although doing so might be slightly
faster for fast-import, it tends to increase the complexity of the frontend
code considerably.</p>
<p>The branch LRU builtin to fast-import tends to behave very well, and the
cost of activating an inactive branch is so low that bouncing around
between branches has virtually no impact on import performance.</p>
<h3>Handling Renames</h3>
<p>When importing a renamed file or directory, simply delete the old
name(s) and modify the new name(s) during the corresponding commit.
Git performs rename detection after-the-fact, rather than explicitly
during a commit.</p>
<h3>Use Tag Fixup Branches</h3>
<p>Some other SCM systems let the user create a tag from multiple
files which are not from the same commit/changeset.  Or to create
tags which are a subset of the files available in the repository.</p>
<p>Importing these tags as-is in Git is impossible without making at
least one commit which &#8220;fixes up&#8221; the files to match the content
of the tag.  Use fast-import's <tt>reset</tt> command to reset a dummy branch
outside of your normal branch space to the base commit for the tag,
then commit one or more file fixup commits, and finally tag the
dummy branch.</p>
<p>For example since all normal branches are stored under <tt>refs/heads/</tt>
name the tag fixup branch <tt>TAG_FIXUP</tt>.  This way it is impossible for
the fixup branch used by the importer to have namespace conflicts
with real branches imported from the source (the name <tt>TAG_FIXUP</tt>
is not <tt>refs/heads/TAG_FIXUP</tt>).</p>
<p>When committing fixups, consider using <tt>merge</tt> to connect the
commit(s) which are supplying file revisions to the fixup branch.
Doing so will allow tools such as <a href="git-blame.html">git-blame(1)</a> to track
through the real commit history and properly annotate the source
files.</p>
<p>After fast-import terminates the frontend will need to do <tt>rm .git/TAG_FIXUP</tt>
to remove the dummy branch.</p>
<h3>Import Now, Repack Later</h3>
<p>As soon as fast-import completes the Git repository is completely valid
and ready for use.  Typically this takes only a very short time,
even for considerably large projects (100,000+ commits).</p>
<p>However repacking the repository is necessary to improve data
locality and access performance.  It can also take hours on extremely
large projects (especially if -f and a large --window parameter is
used).