Tech and Media Labs
This site uses cookies to improve the user experience.


RionWriter

Jakob Jenkov
Last update: 2019-09-21

The RION Ops RionWriter class, com.nanosai.rionops.rion.write.RionWriter, enables you to write RION fields easily and efficiently. In this RionWriter tutorial we will see how to write RION with the RionWriter.

Create a RionWriter

You create a RionWriter using its constructor. Here is an example of creating a RION Ops RionWriter:

RionWriter rionWriter = new RionWriter();

Set Destination

The RionWriter writes the RION fields into a byte array. This byte array is referred to as the destination for the RionWriter. Therefore you must set the destination byte array of the RionWriter before you start writing RION fields with it. You set the destination byte array of the RionWriter using the setDestination() method. Here is an example of setting the destination byte array of a RionWriter:

// 1MB byte array
byte[] destination = new byte[1024 * 1024];

RionWriter rionWriter = new RionWriter();

rionWriter.setDestination(destination, 0);

Notice that the setDestination() method takes 2 parameters. The first parameter is the destination byte array. The second parameter is the offset into the destination byte array the RionWriter should start writing.

Full RionWriter Write Example

Here is a full RionWriter write example that shows how to write RION fields with the RionWriter :

byte destination = new byte[1024 * 1024];
int  startOffset = 0;

RionWriter rionWriter = new RionWriter();
rionWriter.setDestination(destination, startOffset);

rionWriter.writeInt64(123);
rionWriter.writeFloat64(987.654);
rionWriter.writeUtf8("Rion Ops Example");

int totalBytesWritten = rionWriter.index - startOffset;

This example writes 3 RION fields to the destination byte array. A 64 bit integer, a 64 bit floating point, and a UTF-8 text field. The last line of the example calculates how many bytes the RionWriter has written to the destination byte array. You need this information if you want to read the RION fields again with a RionReader, or if you want to write them to disk, network etc.

The RionWriter index field is updated after every single write of a RION field to the destination byte array.

Write Bytes

The RionWriter contains a method named writeBytes() which can be used to write a RION Bytes field to the destination byte array. The writeBytes() takes a byte array as parameter. The content of this byte array will be written into the RION Bytes field. Here is a Java example of writing a RION Bytes field from a byte array using the RionWriter writeBytes() method:

rionWriter.writeBytes( new byte[]{1,2,3,4,5} );

The writeBytes() method also exists in a version that takes a source offset and source length which specifies from what offset and how many bytes forward in the source byte array to write to the RION Bytes field in the destination byte array. Here is an example that only writes part of the source byte array:

rionWriter.writeBytes( new byte[]{1,2,3,4,5,6}, 2, 3);

This example will write the bytes from offset 2 (value = 3) and 3 bytes forward (until and including value 5) as a RION Bytes field in the destination byte array.

Write Boolean

The RionWriter method writeBoolean() writes a boolean value (true / false) as a RION Boolean field to the destination byte array. Here is a Java example of writing a RION Boolean field using the RionWriter writeBoolean() method:

rionWriter.writeBoolean(true);

Write Int64

The RionWriter method writeInt64() will write a 64 bit integer as a RION IntPos or IntNeg field to the destination byte array. If the int64 value is positive, a RION IntPos field will be written. If the value is negative, a RION IntNeg field will be written. Here are two examples of writing int64 values (long values) as RION IntPos and IntNeg fields to the destination byte array, using the RionWriter writeInt64() method:

rionWriter.writeInt64(  1234567890 );
rionWriter.writeInt64( -1234567890 );

Write Float32 and Float64

The RionWriter methods writeFloat32() and writeFloat64() writes a 32 bit and 64 bit floating point number as a RION Float field to the destination byte array. Here are two examples of writing a 32 bit and 64 bit floating point number as RION Float fields to the destination byte array, using the RionWriter writeFloat32() and writeFloat64() methods:

rionWriter.writeFloat32( 123.456F );
rionWriter.writeFloat64( 123456.7890123D );

Write UTF-8

The RionWriter method writeUtf8() writes a Java String as a RION UTF-8 or UTF-8-Short field to the destination byte array. Whether a UTF-8 or UTF-8-Short field is written depends on the length of the String. Here is an example of writing a RION UTF-8 field to a byte array using the RionWriter writeUtf8() method:

rionWriter.writeUtf8("A string to write to UTF-8");

Write UTC

The RionWriter method writeUtc() method writes a date and time in the shape of a Java Calendar as a RION UTC field (UTC date-time) to the destination byte array. Here is an example of writing a Java Calendar as a RION UTC field to the destination byte array using the RionWriter writeUtc() method:

GregorianCalendar calendar = new GregorianCalendar();

rionWriter.writeUtc(calendar, 9);

Notice that the writeUtc() method takes 2 parameters. The first parameter is the Java Calendar instance with the date-timestamp to write. The second parameter tells the "format" of the RION UTC field. Here is what these numbers mean in terms of UTC date-time format:

2 = Year
3 = Year + Month
4 = Year + Month + Day
5 = Year + Month + Day + Hour
6 = Year + Month + Day + Hour + Minutes
7 = Year + Month + Day + Hour + Minutes + Seconds
9 = Year + Month + Day + Hour + Minutes + Seconds + Milliseconds

Write Composite RION Fields

RION has three field types that can contain other RION fields nested inside them. These field types are:

  • RION Array
  • RION Table
  • RION Object

Writing composite RION fields is a bit more complex than writing atomic field fields. The reason for that is, that RION fields use a Type-Length-Value (TLV) encoding. The complexity comes from having to write the length of the field already in the beginning of the field. If you don't know at the time of writing the beginning of a RION Object field how many bytes its nested fields will take up, you cannot correctly write the length of the RION Object field at this time.

The solution is to reserve a number of bytes for the length in the beginning of the composite RION field, and when you are done writing all its nested fields, jump back up to these bytes and set the length correctly.

Obviously, in order to be able to reserve enough length bytes for a composite RION field when you start writing it, you must know its maximal possible length. For instance, if you know the field value will never exceed 256 bytes, you only need to reserve 1 byte to represent its length. Similarly, if the length will never exceed 65,536 bytes, you only need to reserve 2 bytes for the length etc.

Using this method there is a risk of reserving more bytes than what the composite RION field ends up needing, but this is usually not a big overhead compared to the total size of the RION field. If the composite RION field can be up to 16MB in length, you need to allocate 3 bytes to represent its length. If the field ends up only needing 1 byte to represents its length, you will have wasted 2 bytes. However, the advantage of having a TLV format when reading the RION field by far outweighs these wasted length bytes.

The RionWriter class has functionality that helps you reserve length bytes for composite RION fields and jump back up and fill them in correctly later. You will see how that works in the later sections for each of the composite RION fields.

Setting the Nested Field Stack

One of the mechanisms in the RionWriter class that helps you reserve length bytes and set their value correctly later requires an internal stack to keep track of where the reserved length bytes are located. This stack is referred to as a nested field stack. The nested field stack is just an int array. The length of the nested field stack governs how many levels of nesting the given RionWriter will be able to handle. You set the nested field stack via the setNestedFieldStack() method. Here is an example of setting the nested field stack for a RionWriter via its setNestedFieldStack() method:

rionWriter.setNestedFieldStack(new int[16])

This example sets an int array of length 16 (with 16 cells) as nested field stack. This means, that the RionWriter can now handle up to 16 levels of nested RION fields.

Write Object

A RION Object field is a composite RION field which can contain nested RION fields. There are no rules for what fields you nest inside a RION Object field. You can structure the nested fields as you need for the given use case. However, there are two commonly used structures:

  • Key, Value Structure
  • Values Only Structure

Both of these types of RION Object structures are explained in more detail below.

Key, Value Structure

In the key,value structure you nest pairs of RION fields inside the RION Object field. These pairs consists of a RION Key field and the appropriate RION value field for that key. The RION Key field is typically used to represent a field name, or the key used in a Map / dictionary for that value, much in the same way field names are used in JSON. Here is a simple example of what the key, value structure could look like in RION fields:

RION Object
    RION Key
    RION IntPos

    RION Key
    RION UTF-8

    Rion Key
    RION UTC

Each RION Key field is considered the key / field name for the following RION field. In case a RION Key field is followed by another field, the value for the first of these Key fields should be considered to be null. However, it is possible for all RION fields to take the value null, so it is possible to include e.g. a RION UTF-8 field with a null value, rather than omitting it. However, it would be possible to omit it to save bytes.

Using the key, value structure the order of the key, value pairs it typically less important. The RION Key field tells the semantic meaning of the following RION value field. However, if the order of the key, value pairs is fixed, you can hand-write some faster key, value field pair processing than you can if the order is not fixed.

Values Only Structure

In the values only structure, the RION Object field does not contain any RION Key fields. Only the RION value fields are nested inside the RION Object field. Here is a simple example of that the values only structure could look like in RION fields:

RION Object
    RION IntPos
    RION UTF-8
    RION UTC

Using the values only structure will result in smaller RION Object fields (less bytes used). However, you have no Key field to tell what the semantic meaning is of the value field. All you have is the index of the value field. E.g. you may decide that the first nested field is a customer ID, the second is the customer name, the third is the customer birthday etc.

Write Object Begin and End

When writing a RION Object field using the RION Ops RionWriter, you tell the RionWriter when the RION Object begins, how many length bytes to reserve, and when the RION Object ends. Here is an example of marking the beginning and end of a RION Object using the RionWriter :

byte[] rionDestination = new byte[1024];

RionWriter rionWriter = new RionWriter()
        .setNestedFieldStack(new int[16])
        .setDestination(rionDestination, 0);

int reservedLengthBytes = 2; //max 65,536 bytes long Object.

rionWriter.writeObjectBeginPush(reservedLengthBytes);

// write nested RION fields

rionWriter.writeObjectEndPop();

The writeObjectBeginPush() method writes the beginning of the RION Object field, and reserves and reserve as many length bytes as you pass to it as parameter (2 in the example above). The start index of the RION Object field is pushed onto the nested field stack internally for later use.

The writeObjectEndPop() method pops the RION Object field start index off the nested field stack, calculates the total length of the Object field and writes the total Object field length into its reserved length bytes.

There are also versions of the writeObjectBegin() and writeObjectEnd() method that does not use the internal nested field stack. If you want to use these, you will have to keep track of the index of the Object field length bytes yourself, though. This is harder, and makes your code more ugly. But its possible if you prefer.

byte[] rionDestination = new byte[1024];

RionWriter rionWriter = new RionWriter()
        .setNestedFieldStack(new int[16])
        .setDestination(rionDestination, 0);

int objectStartIndex = rionWriter.index;
int reservedLengthBytes = 2; //max 65,536 bytes long Object.

rionWriter.writeObjectBegin(reservedLengthBytes);

// write nested RION fields

int objectLength = rionWriter.index - objectStartIndex;
rionWriter.writeObjectEnd(objectStartIndex, reservedLengthBytes, objectLength);

This avoids the push and pop on the internal nested field stack, but makes your code uglier. Especially if you have multiple layers of nested fields.

Write Key, Value Object

Here is an example of writing a key, value structured RION Object field using the RionWriter:

byte[] rionDestination = new byte[1024];

RionWriter rionWriter = new RionWriter()
        .setNestedFieldStack(new int[16])
        .setDestination(rionDestination, 0);

int reservedLengthBytes = 2; //max 65,536 bytes long Object.

rionWriter.writeObjectBeginPush(reservedLengthBytes);

rionWriter.writeKeyOrKeyShort("customerId");
rionWriter.writeInt64( 123 );  //customerId

rionWriter.writeKeyOrKeyShort("name");
rionWriter.writeUtf8("Jane Doe");  //customer name

rionWriter.writeKeyOrKeyShort("name");
rionWriter.writeUtc(new GregorianCalendar(), 7);  //customer birthdate (today)

rionWriter.writeObjectEndPop();

Write Values Only Object

Here is an example of writing a values only RION Object field using the RionWriter :

byte[] rionDestination = new byte[1024];

RionWriter rionWriter = new RionWriter()
        .setNestedFieldStack(new int[16])
        .setDestination(rionDestination, 0);

int reservedLengthBytes = 2; //max 65,536 bytes long Object.

rionWriter.writeObjectBeginPush(reservedLengthBytes);

rionWriter.writeInt64( 123 );  //customerId
rionWriter.writeUtf8("Jane Doe");  //customer name
rionWriter.writeUtc(new GregorianCalendar(), 7);  //customer birthdate (today)

rionWriter.writeObjectEndPop();

Jakob Jenkov




Copyright  Jenkov Aps
Close TOC