Using the Java Client API

Draft 1.1.

0 Intro

The Freenet reference implementations client library is a flexible and powerful core to build clients around. It shares most of its code (network, crypto, threads) with the reference Freenet node ('Fred'). The flexibility means that it takes a couple of more steps to then just a single call to do an Insert or Request, but that is also what makes it integrateable into anything from a GUI to a converting proxy or remailer.

0.1 Legalities

All this document, except for some of the clearly indicated quotes from the code, is written by Oskar Sandberg. It is covered by the GNU General Public License.

0.2 Corrections, Questions, Flames, etc...

You can email any of the above to me at

perl -e 'print(pack("h*","d6469383d2f6371604e6164616e2b64786e23756"));'

or take them up with the development mailing list.

General Preparation

1.1 The Core

For the client to operate, it needs a client Core to handle the basic services (this is the part that is shared with the Freenet server). Freenet Cores can be very flexible and made to support different transport and presentation protocols, but most clients will only need a basic core to support the standard Freenet protocol over tcp/ip. For efficiency, Cores should be shared between all requests made by a client - it is recommended you create one when you start your client, and keep it alive until the end.

There are two methods Freenet.client.ClientUtil.getTCPCore() that return a Core. The first takes only an integer of the tcp port value that the Core should use for incoming messages as parameter (and if set to 0 that will be selected randomly). The second method also allows you specify a set of parameters, and a Logger. Clients can usually get away without logging or setting parameters, so the first metod is fine, in which case you can skip to 1.2 from here.

1.1.1 Parameters

The Freenet core can take a set of parameters, in the form of Freenet.Params object. A params object can be created by an empty constructor, one that takes a string (which is expected to be in the form "-[=] "), a file (in java properties format, ie "=", EOL delimited, and with # for comments), or as any number of both. Once the object is created, you can set parameter values using the method:

    setParam(String name, String value);

Here are the parameter that are relevant to the client (as taken from the sample.freenetrc file). If you do not understand the meaning of them, you have no reason to change them:

# The number of milliseconds between ticks on the timer
tickerTime=500
# The expected time and standard deviation, in milliseconds, that it takes
# a Freenet node to pass a message. These are used to calculate how long
# the node should wait before assuming that a passed message is lost.
# These are mostly here for debugging reasons - changing them will NOT
# make requests come back faster for you.
hopTimeExpected=12000
hopTimeDeviation=12000
# How long to wait for authentication before giving up (in milliseconds)
authTimeout=30000
# How long to wait to connect to a host before giving up (in milliseconds)
connectTimeout=2000
# How long to wait listen on an inactive connection before closing (if
# reply address is known).
connectionTimeout=300000
# Leave this as "yes" unless you are doing timeout debugging.
doHandshake=yes
# How long to wait for a handshake (in milliseconds)
handshakeTimeout=30000
# How long before a handshake expires (in milliseconds)
handshakeLife=10000000
# Should we use thread-management?  If this number is defined and non-zero,
# this specifies how many inbound connections can be active at once.
maximumConnectionThreads=50
# A list of keytype number class pairs for plugin keytypes
#keyTypes=0101 : Freenet.keys.KHK

Any parameter that is unset will simply be given it's default value. None of these are necessary to make a client work.

1.1.2 Logging

The Freenet Core will log its internal state to an object implementing the Freenet.support.Logger interface. The important method of which is:

    /**
     * Log a message
     * @param source   The source object where this message was generated
     * @param message A clear and verbose message describing the event
     * @param priority The priority of the mesage, one of Logger.ERROR,
     *                 Logger.NORMAL, Logger.MINOR, or Logger.DEBUGGING.
     **/
    public abstract void log(Object source, String message, int priority);

In general, clients should only need to keep track of the internal log of the Core in order to watch for bugs and make helpfully report bugs (and there will be bugs). The priority levels are as follows:

    /** This message indicates an error which prevents correct functionality**/
    public static int ERROR = 3;
    /** A normal level occurance **/
    public static int NORMAL = 2;
    /** A minor occurance that wouldn't normally be of interest **/
    public static int MINOR = 1;
    /** An occurance which would only be of interest during debugging **/
    public static int DEBUGGING = 0;

A good client should probably keep a log of all messages, including debugging, invisibly somewhere to help us when reporting bugs - but it is not necessary. Even ERROR level log entries should be ignorable, as the client will detect problems and send events (see section 5) accordingly when errors occur.

1.2 Address

To create a client, you will also need to give it the address of the Freenet node it should connect to (as you have probably heard, it is strongly recommended that this a local node on the users machine). Freenet.Address objects are protocol independent address wrappers, but since we are dealing with TCP/IP only, you can use the Freenet.client.ClientUtil.getAddress methods to create such an object. The arguments are either a java.net.InetAddress and a port number, a string with the hostname and a port number, or a colon delimited string.

1.3 Client library

Once you have a Core and Address, you can construct a Client library using the constructor:

    public Client(ClientCore core, Address target);

You can use the same Freenet.client.Client object for many requests, but you will have to create a new one if you want to connect to a different node (keep the Core though).

2 Keys

Keys are the backbones of Freenet. To Insert or Request any data, you will need a key for it. Keys also come in several shapes and sizes. This section deals only with creating keys for clients, for documentation on what the key types are an how they work, see the "Bestiary of Keys" document.

2.1 FreenetURIs

Freenet URIs are defined as:

freenet:[KeyType@]KeyValue[,DecryptionKey]

for CHKs, SVKs, and KSKs, though KSKs don't have a DecryptionKey, and don't require a keytype as they are the default. For the SSKs the format is a little different:

freenet:SSK@DocumentName,SubspaceKey

There is a Freenet.client.FreenetURI class that can used to described Freenet URIs. It can be constructed from the string value, and can also return a string in this format from keys. The most important methods to you are:

    getKeyType()

which will tell you what keytype the URI describes, and

    toString()

for outputing keys that are created by the client.

2.1.1 Freenet Base64

For keys that contain binary data (all except KSKs) Freenet uses a modified a version of Base64 modified so as to not contain any chars that have special meanings in URIs, and removing the requirement that the data is a multiple of 4 characters long. The base64 alphabet used is:

  private static char[] base64Alphabet = {
    'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H',
    'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P',
    'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X',
    'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f',
    'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n',
    'o', 'p', 'q', 'r', 's', 't', 'u', 'v',
    'w', 'x', 'y', 'z', '0', '1', '2', '3',
    '4', '5', '6', '7', '8', '9', '~', '-'};

because the client libraries will encode and decode these themselves, you will probably not have to bother with this. If you do, there is a class that does encoding and decoding to and from this format in Freenet.client.Base64.

2.2 CHKs - Content Hash Keys

The object that describes CHKs to the Client is Freenet.client.ClientCHK.

2.2.1 Creating a CHK at Insert.

Because Content Hash Keys are based on the hash of the data, the user doesn't have to (can't) give any value for the key. To create a CHK for inserts, you will have to use the constructor:

    public ClientCHK(InputStream in, long plainLength);

The inputstream should contain the data, but will be consumed when the key is being produced. It cannot be the same stream you give the client later to insert (in other words, you will need to have the data in a file of some kind).

2.2.2 Getting the CHK value.

The full value of the CHK will not be known until after the client libraries have been called. When the data has been inserted, you can call the getURI() method to retun the full URI needed to request the CHK.

2.2.3 Creating a CHK at Request.

When requesting a CHK, the URI will need to contain all the data for the key, so you can simply use the constructor:

    public ClientCHK(FreenetURI uri) throws KeyException;

2.3 SVKs - Signature Verifying Keys

The class that describes SVKs for the Client is Freenet.client.ClientSVK. SVKs will really come into their own when updating is implemented, at which point more documentation on to use them for that will be necessary.

2.3.1 Creating an SVK at Insert.

For inserts, a new private and public key will be generated to create a new SVK. To do this, you need to use the constructor:

    public ClientSVK(Random r) throws KeyException;

Where the java.util.Random implementation is used to generate the key values. You can use Freenet's implementation of the Yarrow cryptographic PRNG by using the randSource field of your Core (see 1.1) as the argument.

2.3.2 Getting the generated SVK value.

The full SVK will not be generated until the client libraries are called, but after the data has been inserted, you can call the getURI() method of the ClientSVK object to return the full URI needed to request the SVK.

2.3.3 Getting the generated private key for an SVK.

When an SVK is generated, a Private key is generated with it. The private key can be used for inserting data in subspaces based on the SVK (see below), and in the future for updating SVK indexed data. After the insert is done, you can get the value of the private key by using the getPrivateKey() method of the ClientSVK object (getPrivateKeyString() will return it base64 encoded).

2.3.4 Creating an SVK at Request.

Like for Requests, SVKs can be created right from the URI when it is requested. Simply used the constructor:

    public ClientSVK(FreenetURI key) throws KeyException;

2.4 KSKs - Keyword Signed Keys

The class that describes KSKs for the Client is Freenet.client.Client.

2.4.1 Creating a KSK for Insert

To create a KSK you will need the keyword value, and a Random implementation (see SVKs above). Use the constructor:

    public ClientKSK(Random r, String keyword) throws KeyException;

2.4.2 Creating a KSK for Request

As for the other keytypes, simply construct it directly from the URI:

    public ClientKSK(FreenetURI key) throws KeyException;

2.5 SSKs - SVK Subspace Keys

Once generated, an SVK can can be used as the root of a subspace, where the private key is needed to insert documents in the subspace, but only the public is needed to request them. SSKs are described to the Client library by the class Freenet.client.ClientSSK.

2.5.1 Creating an SSK for Insert.

When inserting an SSK, you need to the constructor:

    public ClientSSK(Random r, FreenetURI key) throws KeyException

Random is as discussed with SVKs, but the FreenetURI in this case is a special Freenet URI of the form:

freenet:SSK@,

Note that the second part is the private key needed to insert into the subspace (see 2.3.3) not the public subspace key as in the URI that is made public and used to request the data.

2.5.2 Getting the SSK value.

The SSK values URI will be the same as the URI used to construct the ClientSSK, but with the <SVK private key> part replaced by the corresponding public SVK. The value can be retrieved by calling the getURI() method of the ClientSSK object.

2.5.2 Creating an SSK for requests.

Like for all the other keys, when Requesting you can construct the SSK directly from the URI:

    public ClientSSK(FreenetURI key) throws KeyException;

3 Preparing an InsertRequest

Inserts are prepared by calling the prepare method of the Client object (see 1.1 - 1.3) with the paramters:

     * @param htl   The number of HopsToLive to give this message.
     * @param data  A stream of the data to insert.
     * @param dataLength  The number of bytes of data to insert.
     * @param metadata    A stream Freenet standard document metadata.
     * @param metadataLength  The length of the metadata to insert.
     * @param ckey    A ClientKey object to insert by.
     * @param cipherName  The block cipher to use.
     * @param ctBucket    A place to store the CipherText in as it is being
     *                    treated.

The method will return a RequestToken object that can be used to add event listeners and actually start the InsertRequest/DataRequest.

3.1 Hops To Live

An Insert request needs a number of hops to live set, that is the number of nodes that the message should reach. The hops to live of an insert message can generally be a little greater then a request, but no more then 20-25 should be necessary. The current limit of HTL on Freenet is 100.

3.2 Data

To insert something, you will need an InputStream of the data in question. The data will not be read directly from this stream onto Freenet, but rather read as it is being encrypted and hashed, and dumped in a bucket (see below) from which it is inserted (so you can't use this stream as an indication of how much of the data has been inserted).

3.3 Data length

The amount of data to read of the stream in 3.2.

3.4 Metadata

This should be a stream of metadata to insert before the data in Freenet. Observe that this is private metadata, not visible to anybody but those who request the document. Standards from Freenet metadata are documented elsewhere.

If you do not have any metadata, give any stream and set the metadata length to zero.

3.5 Metadata length

The amount of metadata to read off the stream in 2.4. If there is no metadata, set this to zero.

3.6 Key

The key to insert the data by in the form of Freenet.client.ClientKey object. See section 2 for how to generate different kinds of keys.

3.7 Cipher name

This should be the name of the cipher used to encrypt the data. Currently supported ciphers are Twofish and Rijndael, with Rijndael (AES) recommended.

3.8 Ciphertext Bucket

The client library needs to pass over the data twice, once to encrypt and calculate a hash of the ciphertext, and a second time to actually insert the ciphertext. Therefore the ciphertext needs to be dumped somewhere inbetween. To be as flexible as possible, you can give the library any instance implementing the Freenet.support.Bucket interface. To store the data in.

There are two bucket implementations already in Freenet.support. FileBucket is a simple bucket that places the data in a file on the disk, while Cryptbucket will do the same, but encrypt the data a second time with a random key while it is on the disk (this is to protect the user from data recovery from the disk later proving he inserted the data). Note that if you use either of these with a constructor that creates a temporary file, you will need to call Freenet.support.OnExitCleanup.doCleanUp() before exiting for the file to be deleted - if you target is java 1.2, you are better off creating a temporary file yourself with the JFC methods and giving that to File or CryptBucket.

4 Preparing a DataRequest

Requests are prepared by calling the prepare method of the Client object (see 1.1-1.3) with the following parameters:

     * @param htl  The HopsToLive to give the request.
     * @param ck    The key object to request.
     * @param data  A bucket to place the data in
     * @param metaData A bucket to place the metaData in (if there is any)

This method will return a RequestToken to which you can add event listeners and actually start the request.

4.1 Hops to live

This is the amount of nodes that the request should make it to. If Freenet is healthy and the data is available, you should need a value no higher than 15-20 to find the data. The absolute maximum that freenet allows is 100.

4.2 Key

The key to request, as described by a Freenet.client.ClientKey object. See section 2 for information on how to create ClientKey objects.

4.3 Data bucket

The client needs to download all the data before it can verify that it is correct, so it needs a place to store it. For flexibility, you can provide any implementation of Freenet.support.Bucket interface for this purpose.

There are two implementations of Bucket already in Freenet.support, see 3.8 for more information.

4.4 Metadata bucket

As with the data, the client needs to place the metadata somewhere while verifying it, so you must provide a bucket for that too. Metadata is generally small, so it may be more efficient to use a bucket that stores in memory then on disk.

5 ClientEventListeners

When a request (InsertRequest or DataRequest) executes, it will generate events that can be caught to read the status of the execution. Each event will be an implementation of Freenet.client.ClientEvent, and in order to cache them you need to have an object that implements the Freenet.client.ClientEventListener that you can add to the RequestToken generated in step 3 or 4, using:

    public void addEventListener(ClientEventListener cel);

You can add as many event listeners as you please. There is currently one ClientEventListener implemented with the client Library, EventLogger, which will send each events toString() to a Logger object (see 1.1.2).

5.1 Events

The events that the Client library generates are all in the Freenet.client.events package. Each has it's own class, but also a unique number. Here is a list of the events by number and what they mean:

	0	StateReached Event.

This is thrown when the state of the client changes. The state is described by an int, which can be converted to a name with the method:

    Client.stateOf(int state);

The possible states are:

    public static final int INIT = 0;
    public static final int PREPARED = 1;
    public static final int REQUESTING = 2;
    public static final int TRANSFERING = 3;
    public static final int DONE = 4;

    public static final int FAILED = -1;

The state of the client generally progresses, but can regress from TRANSFERING to REQUESTING if the data does not verify. DONE and FAILED are terminal.

	1	SendEvent

A message was sent to Freenet. The getMessageName() method of this class returns the type of message that was sent.

	2	ReceiveEvent

A message was received from Freenet. The getMessageName() method of this class returns the type of message that was received.

	3	ExceptionEvent

An exception occured while executing the client. The most common type of exception are ConnectionExceptions if it fails to connect to the specified address, or NullPointers if you sent a null somewhere where you shouldn't have. The class has a rethrow() method that always throws the exception in question (so you can get a stacktrace).

	4	RestartedEvent

Each Freenet node keeps track of the next, and makes sure that it replies in due time, and that the data it sends back is correct. If a node does not reply to a (data/insert) request, or replies incorrectly, the last good node will start over, and send back a message to the earlier nodes to tell them they can reset their clocks. When such a message reaches the client, a RestartedEvent will occur.

This is what should be happening when a NoReplyEvent (10) occurs. Event 10 more or less means "I (the client) am the last good node, and there is not much I can do".

	5	RequestSuccessfulEvent

This indicates that either the data being requested was found, or the a path to insert the data was found. When this event occurs, the library starts actually transfering the data. This does not indicate that the transaction is complete - in the case of the request the data may be corrupted in which case the network will start over.

	6	RequestFailedEvent

This occurs when a DataRequest or InsertRequest failed gracefully, ie a message was received from the network rejecting the request. The getMessage() method will return the message in question, but only the type will be interesting to most clients.

On Insert:

A "DataReply" means the same thing as CollisionEvent below - data for this key was already found on Freenet.

A "TimedOut" also means that the key was found, but in this case the data was not found (this may seize to be a valid reply on inserts soon since it is insecure - nodes don't have to back up the clame with data to prove it).

A "RequestFailed" means that the InsertRequest went to every node it could reach on all of Freenet before it's hops to live ran out. This should not happen on a network with more than 10 nodes or so - if this happens the users node is most probably disconnected from the main network.

On Request:

A "TimedOut" means that the DataRequest went through the number of nodes you asked, but did not find the data. Increasing the hops to live may help, but it doubtful raising it beyond 25-30 will make any difference at all.

A "RequestFailed" means the same thing as for the InsertRequest.

        7       CollisionEvent

The key was found to already be on Freenet when attempting to insert. For SVKs, KSKs, and SSKs, this could mean other data is already on Freenet, but for CHKs this means the exact data is already on Freenet so the user can just link to/redirect to/use that data instead.

	10	NoReplyEvent

This is generated when no reply is heard from Freenet after a time calculated by a standard formula (90% confidence interval of an expected hop time and deviation of 12 seconds each). If the node you are talking to is working correctly, you should always get a reply within this time (warning: our nodes have been known to not always work correctly). If you fuck with the settings of hopTimeExpected and hopTimeDeviation in section 1.1.1 you can expect to see these.

	16	ErrorEvent

An error occured in the client library. This normally means we fucked up, and after checking the comment and making sure you didn't send anything really weird into the client, this may validate a bug report.

	128	TransferStartedEvent

This event indicates that the transfer of data has started.

	129	TransferedEvent

These events are produced at set intervals while the data is being transfered. The getProgress() method returns the number of bytes read or written thus far.

	130	TransferCompleteEvent

This event indicates that a transfer of data has been completed successfully.

	131	RequestCompleteEvent

This event is produced when all the data you requested has been sucessfully received and the final message accepted. Congratulations ! :-)

	132 	SegmentCompleteEvent

This event indicates that one segment of data has been transfered. Currently, Freenet has up two two segments (metadata and data). Note that this event does *NOT* guarantee that the data in the segment is correct. Only after a RequestCompleteEvent can one assume that the data was intact.

6 Executing the Request

When the you have added any ClientEventListeners (see section 5) to the RequestToken produced in section 3 or 4, you are ready to execute the request. All you need to do is call the

    makeRequest(RequestToken rt)

method of the same Client object that you got on that RequestToken from. The Request will occur asynchronously in another thread, so you all you need to do is wait and watch the events being produced. If you want to wait for the event to finish, you can write a simple EventListener that wakes you up when a StateReachedEvent (see 5.1, Event 0) for either DONE or FAILED occurs, and go to sleep.

	the free network project
	- rewiring the internet -