Add documentation about used backend files

diff --git a/BACKEND b/BACKEND
new file mode 100644 (file)
index 0000000..cde9ef4
--- /dev/null
+++ b/BACKEND
@@ -0,0 +1,95 @@
+$Id$
+
+Most package data is available via sleepycat databases. Those have been
+specifically crafted for efficient lookup, and reasonable times to generate
+them, so that both the daily cronjobs and the actual pagerequests take only a
+moderate amount of CPU and I/O
+
+The only pages that are really generated staticly are the pages listing all
+packages in a given section.
+
+
+This is a brief overview of the available databases:
+
+*********************************************************
+Generated by means of Packages.gz files:
+*********************************************************
+
+| packages_small.db:
+|  key: packagename
+|  value: \0 separated tuples of "archive suite arch component section priority version shortdescription"
+|          (so you can split on spaces in 8 pieces, but need to not split further
+|          because shortdescription can have spaces)
+|          arch can also be 'virtual', with c/s/p/v being undefined then, and
+|          shortdescription being a space-separated list of packages providing
+|          the package that is the key
+  Notes: - maybe add did right before shortdescription?
+         - TODO: make sure for each (archive,suite), newest package is shown
+           first, and all newest versions for each such section is first, so
+           that one can efficiently lookup just the newest entry for a given
+           (archive,suite)
+
+| package_postfixes.db:
+|  key: a postfix string of a package name
+|  value: \0-separated list of prefixes that can preseed this postfix with '^'
+|         instead of the empty string in case a postfix happens to (also?) be
+|         a full package name
+ Note: value can also be \01<decimal number>, meaning there were <decimal
+ number> different packages with that postfix (always more than 100)
+
+| packages_descriptions.db:
+|  key: "packagename version arch"
+|  value: a unique description id, did
+ 
+| descriptions.txt:
+|  on each line:
+|    description with strange characters mangled for proper substring
+|    searching, linenumber being the did
+
+| descriptions.db:
+|  key: did
+|  value: description, first line being short, the rest being long [no
+|            newline transformation]
+
+| descriptions_packages.db:
+|  key: did
+|  value: one or more occurances of: "packagename version arch", separated by \0
+
+| packages_all_$suite.db:
+|  key: "packagename arch version"
+|  value: \0-separated pairs of key\0value items, with key being always
+|         lowercase and having most normal Packages.gz entries, except:
+|         - source: always available, contains straight source package name
+|         - description: has did (description id) only
+|         - archive: notes source archive
+
+| sources_packages.db:
+|  key: sourcepackagename
+|  value: \0 separated tuples of "archive suite package version arch"
+
+  Note: this also comes from the Packages.gz files, and not from Sources.gz
+  files, for accuracy.
+
+*********************************************************
+Generated by means of Sources.gz files:
+*********************************************************
+
+| sources_small.db:
+|  key: sourcename
+|  value: \0 separated tuples of "archive suite component section priority version"
+
+| source_postfixes.db:
+|  key: a postfix string of a source name
+|  value: \0-separated list of prefixes that can preseed this postfix with '^'
+|         instead of the empty string in case a postfix happens to (also?) be
+|         a full package name
+ Note: value can also be \01<decimal number>, meaning there were <decimal
+ number> different packages with that postfix (always more than 100)
+
+| sources_all_$suite.db:
+|  key: "archive suite sourcename"
+|  value: \0-separated pairs of key\0value items, with key being always
+|         lowercase and having most normal Sources.gz entries, except:
+|         - files: \01 separated list of "md5 size filename"
+  Note: different key from packages_all, is that needed?
+