added in v1.76

rclone index

Write static directory listings into a remote.

Synopsis

Write a directory listing, index.html by default, into every directory of remote:path so that it can be browsed when it is served as a static website, for example from an S3 website endpoint, a bucket behind a CDN, or a plain web server.

This works like rclone sync: each run makes the minimum changes needed to bring the listings into line with the files. Listings which haven't changed aren't rewritten, so a run with no changes makes no uploads. The listings never show the listing files themselves.

Note that rclone index treats every file with an output name (index.html by default) in the directories it indexes as its own, and will overwrite it or delete it without checking where it came from. To protect a hand-written index, exclude its directory with --index-exclude.

Outputs

--output NAME=FORMAT says what to write in each directory and can be repeated. NAME is the file name, and FORMAT is one of the built-in formats below or the path of a Go template file.

Format MIME type Contents
html text/html; charset=utf-8 The HTML listing from the rclone serve http template
json application/json JSON in the format of rclone lsjson for the directory
caddy application/json JSON in the format of Caddy's file_server browse

The default is --output index.html=html. The built-in formats are templates, so they can be printed with --print-template FORMAT, copied and changed. Outputs whose name ends in .html or .htm are rendered with Go's html/template and everything else with text/template, which has a json function for safe encoding. The template data is that of rclone serve http with .Static set (see rclone serve http). Outputs are uploaded with the MIME type of the built-in format, or that of the output name's extension for template outputs, since static hosts serve the stored type.

Filters

The normal filters (--include, --exclude, --filter-from, --max-age, ...) control what appears in the listings. An excluded file isn't listed, and an excluded directory isn't listed or indexed.

The --index-include, --index-exclude, --index-filter flags and their -from variants control which directories get listings. They take the same patterns as the normal filters, applied to directory paths, and --index-include implies excluding all other directories. A directory excluded here still appears in its parent's listing, but nothing is written or deleted inside it, so it can have its own hand-made index. --index-max-depth limits how deep listings are written.

Modification times

Listings show the modification time of each file. When using the s3, oracleobjectstorage and swift backends rclone stores the modification time as metadata which isn't returned by the bucket listing, so every object needs a HEAD request, which can mean thousands of transactions per run. Avoid this with one of:

  • --use-server-modtime uses the time the object was uploaded instead, which is what the web server sends as Last-Modified. Recommended for s3 unless the original modification times matter.
  • --no-modtime shows no times at all, which is the cheapest option.

--dir-time sets the time shown for directories:

  • newest (default): the newest modification time of anything below the directory. Note that this means one new file changes the listing of every directory above it.
  • dir: the directory's own modification time. On most file systems writing a listing into a directory changes its time, so this can take more than one run to settle.
  • none: no time.

The listing files are given the newest modification time below their directory, so the Last-Modified header of a listing is meaningful.

Directory links are dir/ by default, which works on hosts that serve index documents: S3 website endpoints, GCS and Azure static websites, nginx and Apache, and R2 custom domains with a URL rewrite rule. For hosts that don't (plain S3 REST URLs, B2 friendly URLs) use --link-index to make directory links dir/index.html instead.

Partial runs

After a sync which changed a known set of files there is no need to walk the whole remote. Tell rclone index what changed and it re-indexes only the directories that could be affected:

rclone index r2:bucket --changed v1.76.0/ --changed version.txt
rclone index r2:bucket --changed-from changes.txt
rclone sync ./public r2:bucket --exclude index.html --combined - | rclone index r2:bucket --changed-combined -
  • --changed PATH names a changed file or directory relative to remote:path and can be repeated. A directory, given with a trailing slash or found to be one, is re-indexed completely.
  • --changed-from FILE reads one path per line exactly as written.
  • --changed-combined FILE reads the --combined report of rclone sync: lines starting with +, -, * and ! are changes and = lines are ignored.
  • Both files accept - for standard input and all three flags can be combined.

A changed file re-indexes its directory and every directory above it, since their times and their listings can change too, and on bucket based storage a directory can cease to exist. Each of those is one directory listing. With --dir-time newest the directories above also need the times of their unchanged subdirectories, which are read from the modification times of those subdirectories' listings at the cost of one small request each. With --use-server-modtime that is the time the listing was written rather than the time of the newest file, so partial runs can drift by the indexing delay until the next full run. --dir-time dir and none need no such lookups.

Directories which are neither named nor above a named path are left alone, as are directories deeper than --max-depth. An empty change list does nothing, and naming the root does a full run. There is no automatic cutover to a full run since that depends on the size of the whole remote: a partial run costs one listing per affected directory, a full run one request per thousand objects on S3-like storage or one per directory elsewhere. The number of directories a partial run will list is logged with -v, and --changed-max-dirs makes it fall back to a full run above that number. A good pattern is a partial run after each upload and a scheduled full run to catch anything missed.

Site icon

The listings carry no icon or branding of their own. To give a site an icon, upload a favicon.ico to the root of the remote, which browsers fetch from there for every page. Exclude it from the listings with --exclude /favicon.ico so it stays on the site but isn't shown.

Deleting listings

On backends which can't have empty directories (S3, B2 and other bucket-based storage) a directory only exists while it has files in it. Once the last file in a directory has been removed its listing is deleted, since the directory only existed because of it. Backends which can have empty directories keep a listing for each empty directory.

Using with sync

A sync from a local build deletes files on the destination which aren't in the source, which includes the listings. Either index the local build before syncing:

rclone index ./public
rclone sync ./public r2:bucket

or exclude the listings from the sync so it neither uploads nor deletes them, then index the remote:

rclone sync ./public r2:bucket --exclude index.html
rclone index r2:bucket

Examples

An S3 website bucket, using the upload times so no object needs a HEAD request:

rclone index s3:my-site --use-server-modtime

HTML and rclone JSON listings, setting Cache-Control on them:

rclone index s3:bucket --output index.html=html --output index.json=json --header-upload "Cache-Control: max-age=300"

Only list release files, and don't write listings under /private:

rclone index r2:bucket --include "*.{zip,deb,rpm}" --index-exclude "/private/**"

A B2 bucket served from its friendly URLs:

rclone index b2:bucket --link-index

See what would change:

rclone index r2:bucket --dry-run -v

Listings which haven't changed aren't normally rewritten, so changing the headers or metadata set on them needs --index-rewrite, which writes every listing a run would render whether it changed or not:

rclone index r2:bucket --index-rewrite --header-upload "Cache-Control: max-age=3600"

rclone index remote:path [flags]

Options

      --changed stringArray              Only re-index directories affected by this changed path (directories end in /)
      --changed-combined stringArray     Read changed paths from a sync --combined report (use - to read from stdin)
      --changed-from stringArray         Read changed paths from file, one per line (use - to read from stdin)
      --changed-max-dirs int             Do a full index if a partial one would list more than this many directories (0 for no limit)
      --dir-time newest|dir|none         How to work out the time shown for a directory (default newest)
  -h, --help                             help for index
      --index-exclude stringArray        Don't write listings in directories matching pattern
      --index-exclude-from stringArray   Read directory exclude patterns from file (use - to read from stdin)
      --index-filter stringArray         Add a rule for which directories get listings
      --index-filter-from stringArray    Read directory rules from a file (use - to read from stdin)
      --index-include stringArray        Only write listings in directories matching pattern
      --index-include-from stringArray   Read directory include patterns from file (use - to read from stdin)
      --index-max-depth int              Only write listings this many directories deep (-1 for no limit) (default -1)
      --index-rewrite                    Write every listing even if it is unchanged
      --link-index                       Link to dir/NAME rather than dir/ for hosts without index documents
      --no-modtime                       Don't show modification times in listings
      --output stringArray               Listing to write in each directory as NAME=FORMAT (default [index.html=html])
      --print-template string            Print the built-in template for FORMAT and exit

Options shared with other commands are described next. See the global flags page for global options not listed here.

Copy Options

Flags for anything which can copy a file

      --check-first                                 Do all the checks before starting transfers
  -c, --checksum                                    Check for changes with size & checksum (if available, or fallback to size only)
      --compare-dest stringArray                    Include additional server-side paths during comparison
      --copy-dest stringArray                       Implies --compare-dest but also copies files from paths into destination
      --cutoff-mode HARD|SOFT|CAUTIOUS              Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
      --ignore-case-sync                            Ignore case when synchronizing
      --ignore-checksum                             Skip post copy check of checksums
      --ignore-existing                             Skip all files that exist on destination
      --ignore-size                                 Ignore size when skipping use modtime or checksum
  -I, --ignore-times                                Don't skip items that match size and time - transfer all unconditionally
      --immutable                                   Do not modify files, fail if existing files have been modified
      --inplace                                     Download directly to destination file instead of atomic download to temp/rename
  -l, --links                                       Translate symlinks to/from regular files with a '.rclonelink' extension
      --max-backlog int                             Maximum number of objects in sync or check backlog (default 10000)
      --max-duration Duration                       Maximum duration rclone will transfer data for (default 0s)
      --max-transfer SizeSuffix                     Maximum size of data to transfer (default off)
  -M, --metadata                                    If set, preserve metadata when copying objects
      --modify-window Duration                      Max time diff to be considered the same (default 1ns)
      --multi-thread-chunk-size SizeSuffix          Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi)
      --multi-thread-cutoff SizeSuffix              Use multi-thread downloads for files above this size (default 256Mi)
      --multi-thread-streams int                    Number of streams to use for multi-thread downloads (default 4)
      --multi-thread-write-buffer-size SizeSuffix   In memory buffer size for writing when in multi-thread mode (default 128Ki)
      --name-transform stringArray                  Transform paths during the copy process
      --no-check-dest                               Don't check the destination, copy regardless
      --no-traverse                                 Don't traverse destination file system on copy
      --no-update-dir-modtime                       Don't update directory modification times
      --no-update-modtime                           Don't update destination modtime if files identical
      --order-by string                             Instructions on how to order the transfers, e.g. 'size,descending'
      --partial-suffix string                       Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
      --refresh-times                               Refresh the modtime of remote files
      --server-side-across-configs                  Allow server-side operations (e.g. copy) to work across different configs
      --size-only                                   Skip based on size only, not modtime or checksum
      --streaming-upload-cutoff SizeSuffix          Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
  -u, --update                                      Skip files that are newer on the destination

Important Options

Important flags useful for most commands

  -n, --dry-run         Do a trial run with no permanent changes
  -i, --interactive     Enable interactive mode
  -v, --verbose count   Print lots more stuff (repeat for more)

Filter Options

Flags for filtering directory listings

      --delete-excluded                     Delete files on dest excluded from sync
      --exclude stringArray                 Exclude files matching pattern
      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
      --exclude-if-present stringArray      Exclude directories if filename is present
      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
      --files-from0 stringArray             Read list of source-file names from file using NUL as separator (use - to read from stdin)
  -f, --filter stringArray                  Add a file filtering rule
      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
      --hash-filter string                  Partition filenames by hash k/n or randomly @/n
      --ignore-case                         Ignore case in filters (case insensitive)
      --include stringArray                 Include files matching pattern
      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
      --max-depth int                       If set limits the recursion depth to this (default -1)
      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
      --metadata-exclude stringArray        Exclude metadatas matching pattern
      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
      --metadata-filter stringArray         Add a metadata filtering rule
      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
      --metadata-include stringArray        Include metadatas matching pattern
      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)

Listing Options

Flags for listing directories

      --default-time Time   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
      --fast-list           Use recursive list if available; uses more memory but fewer transactions

See Also

  • rclone - Show help for rclone commands, flags and backends.