mirror
use mirror to mirror
the complete rsssf.org website
(about 40 000+ .html pages - about 800 MB)
by following (and recording) all internal links.
the command will download (and cache all .html pages
converted to utf-8)
and build-up a local (sqlite) mirror.db (about 20 MB)
with the link structure (ingoing and outgoing) & titles
using a pages and links table.
for more, see /mirror »
prepare
use prepare to download (if not cached or forced)
and convert tables (in .html to .txt)
Usage: prepare/prepare.rb [options] <config slug>
-u, --update turn on update; write to production repo (default: false)
--force force download; do NOT use cached version if available (default: false)
--offline, --cached no downloads; always use cached version (default: false)
pass in a pages config (e.g. eng, de, worldcup, etc.) with a list of table files in a comma-separated values (csv) file.
$ ruby prepare/prepare.rb eng
$ ruby prepare/prepare.rb --force eng ## (force) redownload all
note - the web pages get (by default) cached in ./cache
and the converted tables (in .txt ) get written
to the default outdir ../tables
tip: see https://github.com/rsssf/tables for a public online copy / mirror of converted tables in .txt (preserving the original format).
fmtfix
use fmtfix to convert .txt tables (original format only in .txt)
to .txt pages (applied "autofixes" for football.txt parsing)
Usage: fmtfix/fmtfix.rb [options] <.txt files> or <config slugs>
-u, --update turn on update; write to production repo (default: false)
pass in (i) individual table files e.g eng2010.txt or 34f.txt or (ii) a pages config (e.g. eng, de, worldcup, etc.) with a list of table files in a comma-separated values (csv) file.
$ ruby fmtfix/fmtfix.rb eng2010.txt eng2011.txt
$ ruby fmtfix/fmtfix.rb eng
note - the outdir for pages config default to ./tmp-<slug> e.g. eng becomes ./tmp-eng and so on; for individual table files the outdir defaults to ./tmp-fmtfix
tip: see https://github.com/rsssf/clubs, https://github.com/rsssf/world, for public online copies / mirrors for .txt pages with applied "autofixes" for football.txt parsing (look inside the /pages directories).