Front Page

Explorers
- What is Freenet
- Donate!
- People
- Publicity
- Philosophy
- Translations

Users
- Download
- ScreenShots
- Documentation
- FAQ
- Third Party Tools
- Report Bug
- Mailing Lists

Authors
- Publishing

Developers
- Papers
- Architecture
- Browse CVS
- Public Area
- About This Site

Login & Edit

Hosted by
SourceForge


Freenet Metadata Spec

Author: Adam Langley, Eric Norige


Table of Contents

Overview
Part Spec
Redirect [CDC]
DateRedirect [CDC]
SplitFile [CDC]
Info [MC]
ExtInfo [MC]
Examples
Handling Other Commands

Abstract

This spec is for client metadata. The purpose of this spec is to provide convenient functionality for building web sites inside freenet.

Overview

Metadata = Version command, followed by set of Parts
Part = name of Part, followed by a number of fields, followed by EndPart? (or End for the last Part)
Version command = A Part with the name "Version" and a field called "Revision" of type number. This is 1 for the current spec. Following that an optional "Encoding" field allows for all non-version parts to be gzipped.

More formally

<Metadata> = <Version command> <Parts>* <LastPart> <Arbitrary Metadata>
<Version command> = "Version\n" "Revision=1\n" <Optional: "Encoding=gzip\n"> "EndPart\n"
<Part> = "Document\n" <Field>* "EndPart\n"
<LastPart> = "Document\n" <Field>* "End\n"
<Field> = <Key> "=" <Value> "\n"

Example 1. Abstract Metadata Format

Version
   Revision=1
   EndPart?
   Document
   Key=value
   EndPart?
   Document
   Key1=value1
   Key2=value2
   End
      

Note

"//" in a URI is reserved for metadata processing. This means that the MSK@.....// format is gone. Document Names are the string which comes after the "//". Each Part should have a field with key "Name". The Part whose Name matches the Document Name should be acted upon.

The order of keys in a Part is not important, and if the same Key appears twice in a part, only the last one's value is used.

Note

All numbers are base 16

Control Document Commands are denoted [CDC] and metadata commands as [MC]

There may only be one section with the same Name.

Part Spec

Redirect [CDC]

Redirect.Target=<URI:>

The Client should redirect to the given URI.

DateRedirect [CDC]

[DateRedirect.Increment=<number: time-grain size in seconds, default=15180 (one day)>]
[DateRedirect.Offset=<number
time-grain in seconds since unix epoch (January 1, 1970) to start increments, default = 0>] DateRedirect?.Target=<URI:>
  • The date format is in UNIX format - seconds from the epoch

  • Only applies to KSKs and SSKs

  • The date must be taken UTC (TZ offset = 0)

The client should take the current time (GMT) and work out the last member of the series of times (offset, offset + i, offset + 2i, offset + 3i, ...) which occurred. The client then replaces the part of the URI after the final slash (/string) with <DATE>-string where <DATE> is the hex encoded number of seconds since the epoch.

Note

"freenet:" is speced otherwise someone could do something like "http://..."

In the case of KSKs, the human readable part is the whole key, so freenet:KSK@style becomes freenet:KSK@3b4cf86e-style

In the case of SSKs, the human readable part is the document name, and freenet:SSK@aabbccddee/style becomes freenet:SSK@aabbccddee/3b4cf86e-style

SplitFile [CDC]

SplitFile?.Size=<hex file size>
SplitFile?.BlockCount=<hex no. of data blocks>
[SplitFile.CheckBlockCount=<hex. no of check blocks>]
   SplitFile?.Block.<n>=<URI>
[SplitFile.Graph.<x>=a,b,c...]
      
  • 0 <= n <= 65535

  • n must be sequential, starting with 1, in the metadata part

  • n is represented in base 16

The document is made up of a number of pieces, allowing swarming.

Note

thanks much to thelema, oierw, mjr and others for this

Size Required

This defines the final size of the original file.

BlockCount Required

This defines the number of pieces of data that there are.

CheckBlockCount Optional

This defines the number of check pieces that there are. If Omitted, should default to 0.

Block.<n> Required

These are the block URIs, most likely CHKs. These must be numbered 1 to BlockCount?+CheckBlockCount. The first BlockCount? blocks are the data blocks, and the next CheckBlockCount? are the check blocks.

From the above, a client can start a swarmed download of the file. Redundant splitting is optional, and information is below.

Graph.<x> Optional

For each check Block.<x>, there must be a Graph.<x> listing the data blocks that check block derives from. Graph entries for 1..BlockCount should not be given, but for BlockCount?+1..BlockCount+CheckBlockCount must be given. A check block may also be derived from other check blocks, but only lower numbered ones.

Info [MC]

[Info.Format=<string: MIME-type>]
[Info.Description=<string
freeform>]

The Info.* namespace is reserved for Dublin Core metadata. Prepend "Info." to the keys you want to use to prevent collisions. See http://www.freenetproject.org/doc/infometadata.html for details.

Format is the proper place to put the document's mime type. Description is a Plain description of this data, not an abstract or TOC.

ExtInfo [MC]

[ExtInfo.Trailing=yes]
   [ExtInfo.URI=<URI>]
      

If ExtInfo?.Trailing is set to yes, the metadata for this file will include all data after the final "End" in the control document. If the ExtInfo?.URI parameter exists, the contents of the URI pointed to should be included in the metadata for the current document.

Examples

Example 2. Pseudo Website

Version
    Revision=1
    EndPart?
    Document
    Redirect.Target=CHK@aabbccddee
    EndPart?
    Document
    Name=split
    SplitFile?.Size=102400
    SplitFile?.BlockCount=3
    SplitFile?.Block.1=freenet:CHK@aabbccddee1
    SplitFile?.Block.2=freenet:CHK@aabbccddee2
    SplitFile?.Block.3=freenet:CHK@aabbccddee3
    Info.Format=text/plain
    EndPart?
    Document
    Name=date-redirect
    DateRedirect?.Increment=93a80
    DateRedirect?.Offset=a8c0
    DateRedirect?.Target=SSK@aabbccddee/something
    End
      

This would declare a sort of website. Assume it is inserted under freenet:SSK@aabbccddee/mysite. Accessing freenet:SSK@aabbccddee/mysite or freenet:SSK@aabbccddee/mysite// would cause the first redirect (without a Name) to be processed.

If freenet:SSK@aabbccddee/mysite//split were accessed the SplitFile? section would be processed, as would the Info section. This would (hopefully) swarm a file, with some configurable concurrency. None of the CHKs being swarmed need any metadata because it's included in the control document.

If freenet:SSK@aabbccddee/mysite//date-redirect were accessed the DateRedirect? section would be processed. This would redirect to some other URI.

Example 3. TrailingInfo? example

Version
    Revision=1
    EndPart?
    Document
    Redirect.Target=CHK@aabbccddee
    ExtInfo?.Trailing=yes
    EndPart?
    Document
    Name=doc1
    Redirect.Target=CHK@aabbccddee1
    Info.Format=text/plain
    ExtInfo?.Trailing=yes
    EndPart?
    Document
    Name=doc2
    DateRedirect?.Target=SSK@aabbccddee/something
    ExtInfo?.URI=CHK@eeddccbbaa
    End
    <XML blah blah>
blah blah
    </XML>

This describes the same website as above, but with metadata in a trailing field. It's completely reasonable for multiple documents to share the same TrailingInfo? metadata. (Since only one needs to be processed, this shouldn't be a problem)

Handling Other Commands

****This section is Deprecated pending further revision of the spec****

Other commands may be inserted if they follow the general structure of metadata commands set out above.

These commands may set an Importance header of type string. "Required" means that the client should ALWAYS stop processing if it doesn't understand this extended command. "Informational" means that the client should NEVER stop processing if it doesn't understand this extended command. "Optional" means it's up to the client to decide.

If not given, Importance defaults to Informational.

Since a client will process, for a given Name, the first command it understands, you can do the following:

Note

WaitRedirect is an example of an extended command, not a speced command

Example 4. An Extended Command

WaitRedirect?.Time=5
   WaitRedirect?.Target=CHK@gargargargar
   Redirect.Target=freenet:CHK@gargargargar
      

The WaitRedirect? will override the Redirect, if the client understands WaitRedirects?.

-- SebastianSpaeth - 24 Feb 2002