Import Upstream version 0.7.2

[jigdo.git] / doc / jigdo-file.1

.\" This manpage has been automatically generated by docbook2man 
.\" from a DocBook document. This tool can be found at:
.\" <http://shell.ipoline.com/~elmert/comp/docbook2X/> 
.\" Please send any bug reports, improvements, comments, patches, 
.\" etc. to Steve Cheng <steve@ggi-project.org>.
.TH "JIGDO-FILE" "1" "10 July 2005" "" ""

.SH NAME
jigdo-file \- Prepare files for Jigsaw Download (distribution of huge files, e.g. CD images).
.SH SYNOPSIS

\fBjigdo-file\fR \fB \fI COMMAND\fB
\fR [ \fB--image=\fIcdrom.iso\fB\fR ] [ \fB--jigdo=\fIcdrom.jigdo\fB\fR ] [ \fB--template=\fIcdrom.template\fB\fR ] [ \fB--force\fR ] [ \fBMORE OPTIONS\fR ] [ \fB\fIFILES\fB\fR\fI ...\fR | \fB--files-from=\fIf\fB\fR ]
 \fBCommon COMMANDs: make-template,
make-image, verify\fR

.SH "DESCRIPTION"
.PP
Jigsaw Download, or short jigdo, is a scheme developed
primarily to make it easy to distribute huge filesystem images
(e.g. CD (ISO9660) or DVD (UDF) images) over the internet, but it
could also be used for other data which is awkward to handle due
to its size, like audio/video files or large software
packages.
.PP
jigdo tries to ensure that the large file (always called
\fIimage\fR from now on) is downloaded in small
parts which can be stored on different servers. People who want to
download the image do so by telling the \fBjigdo\fR(1) \fB(NOT IMPLEMENTED YET)\fR
download tool to process one `\fI\&.jigdo\fR\&' file;
using it, \fBjigdo\fR downloads the parts and
reassembles the image. \fBjigdo-file\fR is used to
prepare the files for download.
.PP
What makes jigdo special is that the parts that are used to
reconstruct the image can have any size and content - they only
need to be contained in a contiguous region anywhere in the
image.
.PP
For example, if you wish to distribute an ISO9660 image
which contains a snapshot of an FTP server, you can instruct
\fBjigdo-file\fR to prepare the download data in such
a way that when people use \fBjigdo\fR to download
the image, \fBjigdo\fR actually fetches the
individual files from the FTP server and assembles them into an
exact copy of your image - during the download! (If the image is
not a filesystem dump, you can use
\fBsplit\fR(1) to create the small parts that the image will be
reassembled from.)
.PP
You are completely free to choose where the individual parts
of the image are stored: They may be in entirely different
directories on different servers (e.g. because of
storage/bandwidth constraints), but this is invisible to the
people downloading your image. The information about available
servers only needs to be added to the
`\fI\&.jigdo\fR\&' file by you before distributing
it.
.PP
The `DETAILS' section below contains technical details on
how jigdo works. The `EXAMPLES' section lists a number of common
scenarios and may help you to get an idea of what jigdo is useful
for.
.SH "OPTIONS"
.PP
Many options are specific to a particular
\fICOMMAND\fR; the ones below are general or
used by several commands. Further options are listed below with
the individual commands. All options are silently ignored if they
are not applicable to the current command. For any
\fIBYTES\fR parameters to options, you can
append one of the letters `k', `M' or `G' to the amount you
specify, to indicate kilobytes, megabytes or gigabytes.
.TP
\fB-h --help\fR
Output short summary of commands and options.
.TP
\fB-H --help-all\fR
Output complete summary of commands and options.
.TP
\fB-v --version\fR
Output program version.
.TP
\fB-i --image=\fIcdrom.iso\fB\fR
Specify location of the file containing the image. The
image is the large file that you want to distribute.
.TP
\fB-j --jigdo=\fIcdrom.jigdo\fB\fR
Specify location of the Jigsaw Download description
file. The jigdo file is a human-readable file generated by
\fBjigdo-file\fR, to which you add information
about all the servers you are going to upload the files to.
\fBjigdo\fR will download this file as the
first step of retrieving the image.
.TP
\fB-t --template=\fIcdrom.template\fB\fR
Specify location of the image `template' file. The
template file is a binary file generated by
\fBjigdo-file\fR, it contains information on
how to reassemble the image and also (in compressed form)
all the data from the image which was not found in any of
the parts.

Depending on the command, each of these three files is
used sometimes for input, sometimes for output. If the file
is to be used for output for a particular command and the
output file already exists, \fBjigdo-file\fR
exits with an error, unless \fB--force\fR is
present.

In most cases, you will only need to specify one out
of \fB-i\fR \fB-j\fR
\fB-t\fR, because any missing filenames will be
deduced from the one you specify. This is done by first
stripping any extension from the supplied name and then
appending nothing (if deducing \fB--image\fR),
`\fI\&.jigdo\fR\&' or
`\fI\&.template\fR\&'.
.TP
\fB-r --report=default|noprogress|quiet|grep\fR
Control how verbose the program is, and what format
the output has: \fBnoprogress\fR is the same as
\fBdefault\fR except that no `x%
done\&' progress messages are printed.
\fBquiet\fR restricts the output to what is
absolutely necessary, mostly error messages.
\fBgrep\fR is only different from
\fBdefault\fR for the
\fBmake-template\fR command: It enables output
in a simple `\fI<offset>
<file>\fR\&' format which is useful when
searching for binary files in other binary files.
.TP
\fB-f --force\fR
Overwrite existent output files without
complaining.
.TP
\fB--no-force\fR
\fBThis is the default.\fR Refuse to
overwrite existent output files.
.TP
\fB-c --cache=\fIjigdo-cache.db\fB\fR
\fBjigdo-file\fR usually needs to read
the entire contents of all the
\fIFILES\fR you specify. If you use it
repeatedly (e.g. because you make a new CD image available
daily), caching the file information will increase the
program's speed significantly. The cache file is
automatically created if it is not yet present. Data is
usually both read from and written to it.
.TP
\fB--no-cache\fR
\fBThis is the default.\fR Do not use a
cache.
.TP
\fB--cache-expiry=\fISECONDS\fB\fR
Set maximum age of cache entries. Any entries older
than this will be removed from the cache. The default is 30
days. You can append one of the letters `h', `d', `w', `m',
`y' to denote hours, days, weeks, months or years,
respectively. A value of `0' or `off' disables expiry, so
that all entries will stay in the cache forever. See the
section `CACHE FILES' below for more information.
.TP
\fB--readbuffer=\fIBYTES\fB\fR
Set size of internal buffers. The default is 128k - if
you have a fast disc, increasing this value may make
\fBjigdo-file\fR faster, but in general,
changing it is not necessary.
.TP
\fB--md5-block-size=\fIBYTES\fB\fR
\fBUninteresting internal parameter.\fR
Set size of blocks into which files are subdivided. The
default is 128k. If you change it, any cache file will have
to be regenerated. Internally, \fBjigdo-file\fR
may choose to use a slightly larger or smaller value.
.TP
\fB-T --files-from=\fIfile\fB\fR
Read file and directory names from the specified file.
If \fIfile\fR is `-', read names from
standard input. Each line in the file is taken as a name, so
the names may contain spaces, but not newline characters. An
empty line causes \fBjigdo-file\fR to stop
reading from the file.

\fBfind\fR(1) is a powerful tool for generating file
lists, but make sure to use `\fBfind -type
f\fR\&' if possible - otherwise, if you instruct
\fBfind\fR to output both a filename and a
symlink to that filename, \fBjigdo-file\fR will
read the file contents twice.
.TP
\fB--hex\fR
Output checksums in hexadecimal instead of Base64-like
format. This should not be used with the
\fBmake-template\fR command, because the
resulting `\fI\&.jigdo\fR\&' file violates the
`\fI\&.jigdo\fR\&' file format. Its intended use
is to make \fBjigdo-file\fR more interoperable
with other Unix shell utilities like \fBmd5sum\fR(1)\&.
.TP
\fB--no-hex\fR
\fBThis is the default.\fR Use jigdo's
own Base64-like encoding of checksums.
.TP
\fB--debug[=help|=all|=\fIUNIT,~UNIT...\fB ]\fR
Switch on or off debugging output. Just `--debug' is
equivalent to `--debug=all'. The argument is a
comma-separated list of unit names for which debugging
output is to be enabled, or disabled if the name is preceded
by `~'. The special name `all' means all units. By default,
debugging output is switched off except for the units
`assert' and `general'. The exact list of available units
for which debugging can be switched on depends on whether
jigdo was compiled with debugging support - the list can be
printed with `--debug=help'.
.TP
\fB\fIFILES\fB\fR
Names of files or directories to use as input. These
are the parts that are contained in the image. In case one
of the names is a directory, the program recursively scans
the directory and adds all files contained in it. While
doing this, it follows symbolic links, but avoids symlink
loops.

If one of the filenames starts with the character `-',
you must precede the list of files with `--'. A value of `-'
has \fBno\fR special meaning in this list, it
stands for a file whose name is a single hyphen.
.SH "COMMANDS"
.PP
The command name is the first non-option argument passed to
\fBjigdo-file\fR\&. Most commands have short
abbreviations as well as long names. \fBThe short command
names should not be used in scripts - there may be incompatible
changes to them in the future!\fR
.SS "MAKE-TEMPLATE, MT"
.PP
Reads \fIimage\fR and
\fIFILES\fR, creates
`\fI\&.jigdo\fR\&' and
`\fI\&.template\fR\&'. This is the main functionality
of \fBjigdo-file\fR\&.
.PP
It is possible to specify both \fB--image=-\fR
and \fB--files-from=-\fR\&. In this case, first the
list of files is read from standard input until an empty line is
encountered. Everything following it is assumed to be the image
data. This can be useful if you use \fBmkisofs\fR(1) or similar programs that can output the complete
image on their standard output, because there is no need to
store the image on disc temporarily.
.PP
If a \fIFILES\fR argument contains
the characters `//\&' (Unix) or
`\\.\\\&' (Windows), this has special meaning. In
the final jigdo file that users will download, each of the parts
is referenced in the `[Parts]\&' section with a
URI of the form `Label:some/filename'. (See `FORMAT OF .JIGDO
FILES' below for a detailed description.) The
`[Servers]\&' section gives a mapping of labels
to servers on the internet, with lines like
`Label=http://myserver.org/jigdofiles/'. Using this information,
\fBjigdo\fR will create the final download URI for
the part, `http://myserver.org/jigdofiles/some/filename'.
Specifying `//\&' (or `\\.\\\&')
in a file or directory name serves to `cut off' the names at the
right directory level. For example, if the Unix path of one of
your \fIFILES\fR is `/path/some/filename',
you can tell \fBjigdo-file\fR to cut off after the
`/path' by passing it the argument `/path//some/filename', or
`/path//' if you want the whole directory scanned. The path
names need not be absolute; `somedirectory//' is also
possible.
.TP
\fB--label \fILabel=/path\fB\fR
Specify a name to use as the label name for a path
on disc. (Influences the output jigdo file.) If you used
`//\&' in the
\fIFILES\fR arguments as described
above, \fBjigdo-file\fR will by default pick
label names automatically (`A', `B' etc.). With this
option, you can give labels more meaningful names. Note
that the label name will only be used if one or more
\fIFILES\fR begin with
`/path//'.

Try to use label names that start with uppercase
characters, to disambiguate them clearly from protocol
names like `http', `ftp'.
.TP
\fB--uri \fILabel=http://some.server.org/\fB\fR
By default, using \fB--label\fR as
described above will cause lines of the form
`Label=file:/path/' to be written to the
`[Servers]\&' section of the output jigdo
file. If you want to override the `file:' URI so that the
line reads `Label=http://some.server.org/', you can do so
by specifying \fB--uri\fR along with
\fB--label\fR\&. Giving just \fB--uri
\fILabel=...\fB\fR without the
corresponding \fB--label
\fILabel=...\fB\fR has no
effect, and even if you specify both, an entry is only
added to the `[Servers]\&' section if the
label is referenced by at least one
`[Parts]\&' entry.

The supplied value is not quoted by the program; if
it contains characters such as space or any of the
characters #"'\\ then you must quote it.
(Under Unix, you may need to quote the value twice to also
protect it from the shell, e.g. \\\\\\\\ or
\&'\\\\' to get a single backslash in the
URI.)

The mapping specified with an \fB--uri\fR
option is ignored if it is already present in the output
jigdo file.

Users of the Windows version may notice that the
`\\\&' directory separators are converted
into `/\&' in the `file:' URIs that are
generated by default. This is done to increase
cross-platform compatibility of `file:' - the
\fBprint-missing\fR command of the Windows
version will automatically re-convert the characters when
it prints the URIs. In case you supply your own `file:'
URIs under Windows using \fB--uri\fR, you must
also exchange `/\&' and
`\\\&'.
.TP
\fB-0 to -9\fR
Set amount of compression in the output template
file, from \fB-0\fR (no compression) to
\fB-9\fR (maximum compression). The default is
\fB-9\fR, which can make the template
generation quite slow. By default, the compression
algorithm used is the same as for \fBgzip\fR(1)\&.
.TP
\fB--gzip and --bzip2\fR
Choose between the gzip and bzip2 compression
algorithms. The default is gzip. Bzip2 usually gives
a better compression ratio, but compression is
significantly slower than with gzip.
.TP
\fB--min-length=\fIBYTES\fB\fR
Set minimum length of a part for
\fBjigdo-file\fR to look for it in the image.
The default is 1k. Parts smaller than this will never be
found in the image, so their data will be included in the
template file. The search algorithm used requires such a
minimum length, otherwise template generation could become
extremely slow. If you know for sure that all your
\fIFILES\fR are larger than a certain
amount, you can increase \fBjigdo-file\fR\&'s
speed slightly by specifying the amount with this option.
There is a hard-wired absolute minimum of 256 bytes -
anything lower will silently be set to 256.
.TP
\fB--merge=\fIFILE\fB\fR
Include the contents of
\fIFILE\fR in the output
`\fI\&.jigdo\fR\&' file. The file can contain
data which you want added to the output (for example, a
`[Servers]\&' section with a list of your
servers as entries), or it can be the jigdo file output by
an earlier run of \fBjigdo-file\fR\&.

It is possible to specify the same file for input
with \fB--merge\fR and for output with
\fB--jigdo\fR\&. However, you will also need to
use \fB--force\fR to make the program overwrite
the old version of the jigdo file with the new one.
\fIFILE\fR can be `-' for standard
input.

When \fBadding\fR new information to
the supplied file, \fBjigdo-file\fR will not
insert new lines into the `[Parts]\&'
section if an entry for the same MD5 checksum (but not
necessarily with the same URI!) already exists, and it
will not insert new lines into the
`[Servers]\&' section if a completely
identical entry already exists.

When \fBreading in\fR the existing
\fIFILE\fR, the behaviour is slightly
different: The program \fBpreserves\fR
entries in the `[Parts]\&' section with
identical checksum, but different URIs. For completely
identical entries (same checksum and URI), only one entry
is preserved and the duplicates are removed. The
`[Servers]\&' section is left
untouched.
.TP
\fB--image-section\fR
\fBThis is the default.\fR Causes
\fBjigdo-file\fR to add an
`[Image]\&' section to the
`\fI\&.jigdo\fR\&' file.

As an exception, a new `[Image]\&'
section is \fBnot\fR added if you use
\fB--merge\fR and the file to merge contains an
`[Image]\&' section with a line which
reads `Template-MD5Sum=\&' (end of line
after the `='). In this case, the generated template
data's MD5 checksum value is just added after the `=' of
the first line of this form in the file - no whole new
`[Image]\&' section is appended. This
behaviour is useful because it allows you to pass via
\fB--merge\fR an `[Image]\&'
section with arbitrary content and then have the MD5
checksum automatically added by
\fBjigdo-file\fR\&. The section `FORMAT OF
.JIGDO FILES' below explains the
`[Image]\&' section contents in greater
detail.
.TP
\fB--no-image-section\fR
Do \fBnot\fR include an
`[Image]\&' section in the
`\fI\&.jigdo\fR\&' file. You need to add one
yourself if you use this option. However, doing that is
not easy (you also need to add a
`Template-MD5Sum\&' line with the correct
checksum, or \fBjigdo\fR will complain), so
use of this option is discouraged.
.TP
\fB--servers-section\fR
\fBThis is the default.\fR Causes
\fBjigdo-file\fR to add a
`[Servers]\&' section to the
`\fI\&.jigdo\fR\&' file. This default section
uses `file:' URIs, which allows for immediate reassembly
of the image from the local filesystem, and is also useful
if you want to edit the file manually and replace the
`file:' URIs with other URIs.
.TP
\fB--no-servers-section\fR
Do \fBnot\fR add a
`[Servers]\&' section at the end of the
`\fI\&.jigdo\fR\&' file. Useful e.g. if you are
going to append the section with a script.
.TP
\fB--match-exec=\fISHELLCOMMAND\fB\fR
Whenever a file is found in the image, execute the
supplied command string by passing it to a shell.
\fBjigdo-file\fR sets up a number of
environment variables with information about the file
match. For example, if the file
`\fI/path//a/b/file\fR\&' was found in the
image and `Label:a/b/file' is going to be written to the
`\fI\&.jigdo\fR\&' file:
.RS
.TP 0.2i
\(bu
\fBLABEL\fR="Label" - Name of the label for the file. The example assumes
that `\fB--label\fR
Label=/path\&' was specified by you.
In the absence of such an option, \fBLABEL\fR
will be set but empty.
.TP 0.2i
\(bu
\fBLABELPATH\fR="/path/" - The path corresponding to the label,
or in other words, the prefix of the matched file's
path that will \fBnot\fR appear in the
output `\fI\&.jigdo\fR\&' file. Is set even
without any `\fB--label\fR\&' option present.
Ends with a slash.
.TP 0.2i
\(bu
\fBMATCHPATH\fR="a/b/" - The rest of the path, without the
leafname of the matched file. Is either empty or ends
with a slash.
.TP 0.2i
\(bu
\fBLEAF\fR="file"
- The leafname of the matched file.
.TP 0.2i
\(bu
\fBMD5SUM\fR="lNVdUSqbo2yqm33webrhnw" - The md5sum of the
matched file, in Base64-like format.
.TP 0.2i
\(bu
\fBFILE\fR="/path//a/b/file" - For convenience, the
complete path of the file. The variable is always set
to $LABELPATH$MATCHPATH$LEAF\&.
.RE

Please be careful to correctly quote the string
passed to this option, otherwise your supplied command
will not work with filenames that contain spaces. As an
example, to create a backup of hard links to the matched
files, use the following option:
--match-exec='mkdir -p "${LABEL:-.}/$MATCHPATH"
&& ln -f "$FILE" "${LABEL:-.}/$MATCHPATH$LEAF"'

By default, no command is executed. Use
--match-exec="" to remove a command string which was set
with an earlier use of this option.
.TP
\fB--greedy-matching\fR
\fBThis is the default.\fR Imagine
that your image contains a \fI\&.tar\fR file
which in turn contains another file
\fIx\fR, and that you provide both the
\fI\&.tar\fR and the files inside it on the
command line. When \fBjigdo-file\fR scans the
image, it encounters the beginning of the
\fI\&.tar\fR file, and then the file
\fIx\fR\&.

At this point, a decision must be made: Should the
smaller file \fIx\fR be recorded as
matched, or should it be ignored in favour of the larger
(and thus better) match of the \fI\&.tar\fR
file? Unfortunately, at this point it is not clear
whether there will actually be a full match of the
\fI\&.tar\fR, so by default, the program
prefers the small match.
.TP
\fB--no-greedy-matching\fR
In the case where a large partial match is present
and a shorter match has been confirmed, ignore the small
match. (See the option above.)
.SS "MAKE-IMAGE, MI"
.PP
Reads `\fI\&.template\fR\&' and
\fIFILES\fR, creates
\fIimage\fR (or
`\fIimagename.tmp\fR\&'). Provides a rudimentary
way of reassembling images - \fBjigdo\fR is usually
better suited for this task. However, in contrast to
\fBjigdo\fR, no `\fI\&.jigdo\fR\&' file
is required.
.PP
If the image is to be written to a file (and not to
standard output), it is possible to create the image in several
steps, with several invocations of `\fBjigdo-file
make-image\fR\&', as follows: You first invoke
\fBjigdo-file\fR, specifying as many files as are
available at this time. The program scans the files, and those
that are contained in the image are copied to a temporary file,
whose name is formed by appending `\fI\&.tmp\fR\&' to
the image filename.
.PP
For all further files which could be parts of the image,
you repeat this process. As soon as all parts are present, the
temporary file will be truncated slightly (to delete some
administrative data that \fBjigdo-file\fR appends
at the end) and renamed to the final image name. The possibility
of reassembling the image in several steps is especially useful
for gathering files from removable media, e.g. several older
CDs.
.PP
Scripts using \fBmake-image\fR can detect
whether image creation is complete by checking the exit status:
0 signals successful creation, whereas 1 means that more files
need to be supplied. Other errors result in an exit status of 2
(`recoverable', e.g. file not found) or 3 (non-recoverable, e.g.
write error).
.TP
\fB--check-files\fR
\fBThis is the default.\fR Whenever
any part is copied to the image, re-check its checksum
against the checksum stored in the template. It is
recommended that you leave this switched on, even if it
slows down image creation a bit.
.TP
\fB--no-check-files\fR
Do not check files' checksums when copying them to
the image. This can be safely used when no cache file is
used (which means that files will be written to the image
immediately after being scanned) or the whole image is
checked later with the \fBverify\fR
command.
.SS "PRINT-MISSING, PM"
.PP
Reads `\fI\&.jigdo\fR\&',
`\fI\&.template\fR\&' and (if present)
`\fIimagename.tmp\fR\&', outputs a list of URIs
still needed to completely reassemble the image.
.PP
Together with the \fBmake-image\fR command,
this provides most of the functionality of
\fBjigdo\fR on the command line.
.PP
For each part that is not yet present in the temporary
image file, the file checksum is looked up in the
`[Parts]\&' section of the jigdo file. Any
label in the corresponding entry is then expanded according to
the label definitions in the `[Servers]\&'
section and printed on standard output. \fBjigdo\fR
allows you to specify several alternative locations for each
label in this section, but \fBprint-missing\fR will
only output the first one for each missing part.
.PP
If the checksum cannot be found in the
`[Parts]\&' section (this Should Not Happen
unless you deleted that section), a lookup is instead made for
`MD5Sum:\fI<checksum>\fR\&', just like
with \fBjigdo\fR\&. (Thus, if you want to get rid of
the `[Parts]\&' section, you can do so if you
rename each part to its own checksum.)
.TP
\fB--uri \fILabel=http://some.server.org/\fB\fR
Override the entries in the
`\fI\&.jigdo\fR\&' file for any label with a
URI of your choice. With the example above, a
`[Parts]\&' entry of
`Label:some/filename' will cause the line
`http://some.server.org/some/filename' to be
printed.

The supplied value is not quoted by the program; if
it contains characters such as space or any of the
characters #"'\\ then you must quote it.
(Under Unix, you may need to quote the value twice to also
protect it from the shell, e.g. \\\\\\\\ or
\&'\\\\' to get a single backslash in the
URI.)
.SS "PRINT-MISSING-ALL, PMA"
.PP
Just like \fBprint-missing\fR, this command
outputs a list of URIs still needed to completely reassemble the
image. However, \fBall\fR alternative download
locations are printed instead of just one. In the output, the
URIs for a file are separated from other files' URIs with blank
lines. The \fB--uri\fR option has the same effect as
for \fBprint-missing\fR\&.
.SS "VERIFY, VER"
.PP
Reads \fIimage\fR (presumably
generated with \fBmake-image\fR) and
`\fI\&.template\fR\&', checks for correct checksum of
image.
.PP
The template data does not only contain checksums of the
individual parts, but also of the image as a whole.
\fBmake-image\fR already performs a number of
internal checks, but if you like, you can additionally check the
image with this command.
.SS "SCAN, SC"
.PP
Reads all the \fIFILES\fR and enters
them into the cache, unless they are already cached. The
\fB--cache\fR option must be present for this
command.
.TP
\fB--no-scan-whole-file\fR
\fBThis is the default.\fR This only
causes the first \fB--md5-block-size\fR bytes
of each file to be read. If the cache is used later by
\fBjigdo-file make-image\fR, the rest of the
file will be read once these first bytes are recognized in
the input image.
.TP
\fB--scan-whole-file\fR
Immediately read the entire file contents and store
them in the cache.
.SS "MD5SUM, MD5"
.PP
Reads all the \fIFILES\fR and prints
out MD5 checksums of their contents. This command is quite
similar to \fBmd5sum\fR(1), except that the checksum is output in the
Base64-like encoding which is also used elsewhere by
\fBjigdo-file\fR\&.
.PP
The \fIFILES\fR arguments are
processed in the same way as with the other commands, which
means that recursion automatically takes place for any arguments
that are directories, and that symbolic links are not listed
except when the file(s) they point to are not reachable
directly.
.PP
In the checksum list printed on standard output, only the
part of the filename following any `//\&' (or
`\\.\\\&' on Windows) is printed. Any
\fB--cache\fR will be used for querying files' MD5
checksums and/or writing the checksums of scanned files.
.SS "LIST-TEMPLATE, LS"
.PP
Reads a `\fI\&.template\fR\&' file and outputs
low-level information about the image and all parts contained in
it, including offset, length and checksum.
.PP
You can also use this command with temporary image files
(by specifying something like
\fB--template=imagename.tmp\fR) - in that case, the
output also distinguishes between parts that have been written
to the image and parts that haven't.
.PP
The exact output format may change incompatibly between
different jigdo releases. The following different types of lines
can be output. `have-file' only occurs for
`\fI\&.tmp\fR\&' files, indicating a file that has
already been successfully written to the temporary file:

.nf
in-template \fIoffset-in-image length\fR
need-file \fIoffset-in-image length file-md5sum filestart-rsyncsum\fR
have-file \fIoffset-in-image length file-md5sum filestart-rsyncsum\fR
image-info \fIimage-length image-md5sum rsyncsum-size\fR
.fi
.SH "DETAILS"
.PP
Jigsaw Download was created with the format of ISO9660 CD
images in mind - however, the following also applies to many other
filesystem formats, as well as to `tar' archives and uncompressed
`zip' archives. A CD image contains both information for
organizing the filesystem (header with disc name etc., ISO9660
directory data, data of extensions such as Joliet or RockRidge,
zero padding) and the files contained on the CD. An important
property that jigdo relies on is that each file is stored in one
contiguous section of the image; it is not split into two or more
parts.
.PP
When \fBjigdo-file\fR is given a number of
files that might be contained in an image, it detects whether any
of the files are present using a `rolling checksum' inspired by
the one used by \fBrsync\fR(1)\&. The resulting data is
written to the `\fI\&.template\fR\&' file: If a section
of the image could not be matched (e.g. it was directory
information), the data is compressed and written directly to the
template. However, if a matching file was found, its data is
omitted from the template. Instead, only a reference (an MD5
checksum of the file) is inserted in the template.
.PP
Note that the template data only contains binary data, it
does not contain any filenames or URIs, since it cannot be easily
edited in case any of these values need to be changed. All that
information is stored in the `\fI\&.jigdo\fR\&' file, a
text file to which you can add URLs for your server(s). The jigdo
file provides a mapping for each MD5 checksum to one or more
alternative download locations for the corresponding part.
.PP
Apart from the mapping of MD5 sums to URIs, the jigdo file
also contains an URI pointing to a download location for the
template file. This way, the \fBjigdo\fR download
tool only needs to be given one URI (that of the
`\fI\&.jigdo\fR\&' file) to be able to download and
reassemble the complete image.
.SH "FORMAT OF .JIGDO FILES"
.PP
The overall format of `\fI\&.jigdo\fR\&' files
follows that of `\fI\&.ini\fR\&' files, as also used by
the Gnome and KDE projects for some data. The file is organized
into sections, each of which is preceded by a line reading
`[Sectionname]\&'. Within each section, lines
have the form `Label=Value'. Such lines are also called `entries'
below. All `\fI\&.jigdo\fR\&' files use UTF-8 as their
character encoding.
.PP
Comments are introduced with the `#\&'
character and extend to the end of the line. Whitespace is ignored
at line start and end as well as to the left and right of section
names and the `=\&' in entries. Furthermore, the
jigdo utilities split up the text of the entry value (i.e. the
part after the `=\&') into whitespace-separated
words, much like the Unix shell. Single \&'' and
double "" quotes can be used to prevent that
e.g. URIs containing whitespace are split apart. Similarly,
characters with special meaning (the characters
\&'"#\\ and space/tab) must be quoted with
\\ to appear in the value. As with the shell,
there is a difference between \&'\~\&' and
"\~": Within \&'\~\&',
the characters "#\\ and whitespace lose their
special meaning and become ordinary characters, whereas within
"\~", only the characters
\&'# and whitespace lose their special meaning -
in other words, backslash escapes still work inside
"\~", but not
\&'\~\&'\&.
.PP
`\fI\&.jigdo\fR\&' files can optionally be
compressed with \fBgzip\fR(1)\&. \fBjigdo-file\fR always outputs
uncompressed files, which you can compress yourself.
\fBjigdo-lite\fR supports single uncompressed and
compressed files.
.PP
(Behaviour which may change in the future and which should
not be relied upon: \fBjigdo\fR additionally supports
any number of concatenated plaintext and gzipped parts in the
files - for example, you can compress a
`\fI\&.jigdo\fR\&' file and then add a couple of lines
of uncompressed data to the end.)
.PP
In all cases, the `\fI\&.gz\fR\&' extension
should be removed from the filename - the tools will determine
automatically from the file contents whether a file is compressed
or not.
.PP
Below is a description of the individual section names used
by jigdo.
.SS "JIGDO SECTION"

.nf
[Jigdo]
Version=1.1
Generator=jigdo-file/1.0.0
.fi
.PP
Information about the version of the jigdo file format
used, and the program that generated it. There should be one
such section per `\fI\&.jigdo\fR\&' file.
.SS "IMAGE SECTION"

.nf
[Image]
Filename=\fI"filename for saving on user's disc"\fR
Template=\fI"URI where to fetch template file"\fR
Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g
ShortInfo=\fIsingle-line description of the image (200 characters max.)\fR
Info=\fIlong description (5000 characters max.)\fR
.fi
.PP
The value for the `Template' entry can be either an URL
(absolute or relative to the URL of the jigdo file) or a string
of the form `\fILabel\fR:\fIpathname\fR\&' (\fBUNIMPLEMENTED\fR),
as described below.
.PP
The `Template-MD5Sum' entry is added by
\fBjigdo-file\fR and specifies the MD5 checksum of
the generated `\fI\&.template\fR\&' file. It is used
by \fBjigdo\fR to detect cases where the downloaded
template data is corrupted or belongs to a different
image.
.PP
Unlike other entry values, the values of the
`ShortInfo\&' and `Info\&'
entries are \fBnot\fR split up into words,
instead all quoting is preserved.
.PP
The value of the `Info\&' entry is
special in that \fBjigdo\fR(1) can optionally parse XML markup it contains. If
the markup has errors such as unbalanced/unsupported tags, the
string is displayed literally, without XML parsing. Supported
tags are <b></b> (bold),
<i></i> (italic),
<tt></tt> (typewriter font),
<u></u> (underline),
<big></big> (larger font),
<small></small> (smaller font)
and <br/> (linebreak). Supported
entities include &lt; (`<\&'),
&gt; (`>\&') and
&amp; (`&\&'). Note that the whole
`Info\&' entry must be on one line in the jigdo
file.
.PP
This section may occur multiple times, but all except the
first one will be ignored. This is useful e.g. when creating a
`\fI\&.jigdo\fR\&' file for a DVD image when you
already have `\fI\&.jigdo\fR\&' files for CDs with
the same content: You can simply `[Include]\&'
(see below) the CDs' jigdo files at the end of the DVD jigdo
file, after its `[Image]\&' section.
.SS "PARTS SECTION"

.nf
[Parts]
xJNkjrq8NYMraeGavUpllw=LabelA:part0
GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1
kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2
-J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3
.fi
.PP
All lines in the section, which provides the mapping from
MD5 checksums to URIs, have the same format: On the left side of
the `=\&' the checksum (encoded with a
Base64-like encoding) is given, and on the right a string
corresponding to the part with this checksum; either a complete
URI or a string of the form `\fILabel\fR:\fIpathname\fR\&', which is expanded into
one or more URIs by looking up the definition(s) for the
\fILabel\fR in the
`[Servers]\&' section.
.PP
In case a particular MD5 checksum cannot be found in any
`[Parts]\&' section by
\fBjigdo\fR, the program will perform a lookup for
`MD5Sum:\fI<checksum>\fR\&', e.g. for
`MD5Sum:xJNkjrq8NYMraeGavUpllw\&' if you
deleted the line for `part0' above.
.PP
A checksum appearing multiple times in this section
indicates alternative download locations for the part.
.PP
There may be any number of `[Parts]\&'
sections in the file; they are all considered when looking up
MD5 checksums.
.PP
\fBjigdo-file\fR always puts the
`[Parts]\&' section at the end of the file, and
it even rearranges any file specified with
\fB--merge\fR to have only one such section at the
end. This is done to allow \fBjigdo\fR to display
the information from the `[Image]\&' section
while the rest of that file is still being downloaded.
.SS "SERVERS SECTION"

.nf
[Servers]
LabelA=http://myserver.org/
LabelA=ftp://mirror.myserver.org/
LabelB=LabelC:subdirectory/
LabelC=http://some.where.com/jigdo/
.fi
.PP
All lines in the section, which provides the mapping from
server labels to server locations, have the same format: On the
left side of the `=\&' the label name is given,
and on the right the value to expand the label name to.
.PP
A label name appearing multiple times in this section
indicates alternative download locations for the parts that use
the label in the `[Parts]\&' section. This
notation makes it very easy to add mirrors to the jigdo
file.
.PP
As shown by the example above, the label values may
themselves reference other labels. In this case, the entry
`LabelB:some/path/part2' in the `[Parts]\&'
section will expand to
`http://some.where.com/jigdo/subdirectory/some/path/part2'.
Loops in the label definitions result in undefined behaviour and
must be avoided.
.PP
There may be any number of `[Servers]\&'
sections in the file; they are all considered when looking up
labels. Either of `[Parts]\&' or
`[Servers]\&', but not both, can be omitted
from the jigdo file.
.SS "COMMENT SECTION"

.nf
[Comment]
Any text, except that lines must not begin with `['.
.fi
.PP
All text following a `[Comment]\&' or
`[comment]\&' line is ignored, up to the next
line with a section label.
.SS "INCLUDE DIRECTIVE"

.nf
[Include http://some.url/file.jigdo]
.fi
.PP
Lines of this form cause the content of the specified
jigdo file to be downloaded and parsed just like the main jigdo
file. The effect will be the same as copying the included file's
contents into the file which contains the include
directive. (Exception: Any relative URLs are always resolved
using the URL of the `\fI\&.jigdo\fR\&' file that
contains that relative URL.)
.PP
The URL argument can be an absolute or relative URL. 
Relative URLs are assumed to be relative to the URL of the jigdo
file which contains the include directive. Includes can be
nested, but it is an error to create a loop of include
directives. It is \fBnot\fR possible to use URLs
of the form `\fILabel\fR:\fIpathname\fR\&'.
.PP
The URL cannot be quoted with "". Any
`]\&' characters in the argument must be
escaped as `%5D\&', and any spaces as
`%20\&'.
.PP
Include directives are only supported by
\fBjigdo\fR, they are ignored by
\fBjigdo-lite\fR\&.
.PP
An include directive terminates any previous section, but
it does not start a new one. In other words, a new section must
always be started after the include line,
\fBjigdo\fR does not allow normal entries to appear
below the `[Include]\&'.
.SH "CACHE FILES"
.PP
Any file specified with the \fB--cache\fR option
is used to store information about the
\fIFILES\fR presented to
\fBjigdo-file\fR\&. When querying the cache, a file is
considered unchanged (and the cached data is used) only if
filename, file size and last modification time (mtime) match
exactly. For the filename match, not the entire file name is used,
but only the part following any `//\&', so that
any changes to the part before the `//\&' will
not invalidate the cache.
.PP
Old cache entries are removed from the cache if they have
not been read from or written to for the amount of time specified
with \fB--cache-expiry\fR\&. Entries are
\fBnot\fR immediately removed from the cache if the
file they refer to no longer exists - this makes it possible to
cache information about files on removable media.
.PP
Cache expiry only takes place \fBafter\fR
\fBjigdo-file\fR has done its main work - if any old
entries are accessed before expiry takes place, they will be kept.
For example, if the program is run using the default expiry time
of 30 days, but accesses a cache file with entries generated 2
months ago, then entries in that cache \fBwill\fR
be considered, and only those cache entries that were not needed
during the program run will be expired.
.PP
Due to a peculiarity of the underlying database library
(libdb3), cache files never shrink, they only grow. If a large
number of entries was expired from your cache file and you want it
to shrink, you can either just delete it (of course then
everything will have to be regenerated) or use the utilities
accompanying libdb3 to dump and restore the database, with a
command like `\fBdb3_dump
\fIold-cache.db\fB | db3_load
\fInew-cache.db\fB\fR\&'. For Debian, these programs are supplied in the
package `libdb3-util'.
.PP
If a different \fB--md5-block-size\fR is
specified, the entire file needs to be re-read to update its cache
entry. If a different \fB--min-length\fR is specified,
only the first `md5-block-size' bytes of the file need to be
re-read.
.SH "EXAMPLES"
.SS "PREPARING YOUR CD IMAGE FOR DISTRIBUTION"
.PP
You have created a CD image
`\fIimage.iso\fR\&' from some of the files stored
in the directory `\fI/home/ftp\fR\&' on your
harddisc, which is also available online as `ftp://mysite.org'.
As you don't want to waste space by effectively hosting the same
data twice (once as files on the FTP server, once inside the
image), and you are fed up with users' downloads aborting after
200MB and their restarting the download dozens of times, you
decide to use jigdo. How do you prepare the image for
download?
.PP
In fact, only one command is necessary:
.sp
.RS
.PP
\fBjigdo-file make-template
--image=image.iso --jigdo=/home/ftp/image.jigdo
--template=/home/ftp/image.template /home/ftp// --label
Mysite=/home/ftp --uri
Mysite=ftp://mysite.org/\fR
.RE
.PP
People can now point \fBjigdo\fR at
`ftp://mysite.org/image.jigdo' to download your image. The
template file needs to be accessible as
`ftp://mysite.org/image.template'.
.PP
Note that nothing prevents you from doing the same for an
FTP server that isn't administrated by you - in that case, you
only need to host the `\fI\&.jigdo\fR\&' and
`\fI\&.template\fR\&' files on your own
server/homepage.
.SS "PREPARING AN ARBITRARY LARGE FILE FOR DISTRIBUTION"
.PP
We assume that you have a large file that is not a
filesystem, e.g. `\fImovie.mpeg\fR\&'. Because of
space problems, you want to distribute the data on two
servers.
.PP
In this case, the parts of the image need to be generated
artificially with the \fBsplit\fR command. For
example, to create chunks of 4MB each, use `\fBsplit -b 4m
movie.mpeg part\fR\&'. Copy the resulting files
`\fIpartXX\fR\&' into
two directories `\fI1\fR\&' and
`\fI2\fR\&' that you create, according to how you
want the files distributed between the servers. Next, create the
jigdo and template files with `\fBjigdo-file make-template
--image=movie.mpeg 1// 2//\fR\&'. You will need to edit the
`\fI\&.jigdo\fR\&' file and provide the right URIs
for the two servers that you are going to upload the
`\fIpartXX\fR\&' files
to.
.SS "CUSTOMIZED VERSIONS OF IMAGES"
.PP
Because it is possible to assign a different URI for each
part of an image if necessary, jigdo is very flexible. Only one
example is the possibility of customized versions of images:
Suppose that someone is distributing a CD image, and that you
want to make a few small changes to it and redistribute your own
version. You download the `\fIofficial.iso\fR\&' CD
image with \fBjigdo\fR (passing it the URL of
`\fIofficial.jigdo\fR\&'), write it to CD-R, make
your changes (say, adding files from the
`\fImyfiles\fR\&' directory on your harddisc) and
produce your own version, `\fImyversion.iso\fR\&'.
Next, you instruct \fBjigdo-file\fR to create the
jigdo and template files for your modified image, using the
command
.sp
.RS
.PP
\fBjigdo-file make-template
--image=myversion.iso /mnt/cdrom/ myfiles// --label
My=myfiles/ --uri My=http://my.homepage.net/
--merge=official.jigdo\fR
.RE
while `\fIofficial.iso\fR\&' is mounted under
`\fI/mnt/cdrom\fR\&'. By using
\fB--merge\fR, you have told
\fBjigdo-file\fR to take the contents of
`\fIofficial.jigdo\fR\&', add to it a new
`[Image]\&' section for
`\fImyversion.iso\fR\&' and write the resulting
jigdo file to `\fImyversion.jigdo\fR\&' - so now
`\fImyversion.jigdo\fR\&' offers two images for
download, the original version and your modified version. (If
you do not want it to offer the official version, edit it and
remove the `[Image]\&' section that lists
`\fIofficial.iso\fR\&'.)
.PP
Now you can upload the `\fI\&.jigdo\fR\&' file,
the `\fI\&.template\fR\&' file and also the files in
`\fImyfiles\fR\&' to `http://my.homepage.net/'.
Thus, for people to download your modified image, you do
\fBnot\fR need to upload the complete image
contents to your web space, but only the changes you
made!
.PP
(In case you only made very few changes, you could also
omit the `myfiles' parameter in the command above, then all your
changes end up in the new template file.)
.SS "COMBINING MANY JIGDO-MANAGED IMAGES INTO ONE"
.PP
It is also no problem to combine data from several sources
that use jigdo. For example, if of five different and unrelated
servers each one distributes a different CD image via jigdo, you
can create a customized DVD image that contains the data from
all these CDs. When people use \fBjigdo\fR to
download your image, the individual files on the DVD are fetched
from the same sources as the original CDs.
.PP
Consequently, even though you will be distributing a 3.2GB
file via your web space, the actual amount of data that is
stored on your server will only be in the order of several
MBs.
.SH "BUGS"
.PP
For certain contents of one of the input files, most notably
a sequence of zero bytes longer than \fB--min-length\fR
at the start of the file and an area of zeros preceding the file
data in the image, \fBjigdo-file make-template\fR may
fail to find the file in the image. Unfortunately, this
restriction cannot be avoided because the program could become
very slow otherwise. If you use the \fB--debug\fR
option, all instances of \fBjigdo-file\fR discarding
possible matches are indicated by lines containing the word
`DROPPED\&'.
.PP
In fact, not only all-zeroes files trigger this behaviour,
but also files which contain at their start a long sequence of
short identical strings. For example, both a file containing only
`a\&' characters and one containing
`abcabcabcabc\&...' are problematic.
.SH "SEE ALSO"
.PP
\fBjigdo\fR(1) (NOT YET IMPLEMENTED),
\fBjigdo-lite\fR(1),
\fBjigdo-mirror\fR(1),
\fBsplit\fR(1) (or `\fBinfo split\fR\&'),
\fBfind\fR(1) (or `\fBinfo find\fR\&'),
\fBmkisofs\fR(1),
\fBmd5sum\fR(1)
.SH "AUTHOR"
.PP
Jigsaw
Download <URL:http://atterer.net/jigdo/> was written by Richard Atterer
<jigdo atterer.net>, to make downloading of CD ROM
images for the Debian Linux distribution more convenient.