RionWriter
Jakob Jenkov |
The RION Ops RionWriter class, com.nanosai.rionops.rion.write.RionWriter,
enables you to write RION fields easily and efficiently. In this RionWriter tutorial we will
see how to write RION with the RionWriter.
Create a RionWriter
You create a RionWriter using its constructor. Here is an example of creating a RION Ops
RionWriter:
RionWriter rionWriter = new RionWriter();
Set Destination
The RionWriter writes the RION fields into a byte array. This byte array is referred to as the
destination for the RionWriter. Therefore you must set the destination byte array of the
RionWriter before you start writing RION fields with it. You set the destination byte array
of the RionWriter using the setDestination() method. Here is an example of setting
the destination byte array of a RionWriter:
// 1MB byte array byte[] destination = new byte[1024 * 1024]; RionWriter rionWriter = new RionWriter(); rionWriter.setDestination(destination, 0);
Notice that the setDestination() method takes 2 parameters. The first parameter is the
destination byte array. The second parameter is the offset into the destination byte array the
RionWriter should start writing.
Full RionWriter Write Example
Here is a full RionWriter write example that shows how to write RION fields with the
RionWriter :
byte destination = new byte[1024 * 1024];
int startOffset = 0;
RionWriter rionWriter = new RionWriter();
rionWriter.setDestination(destination, startOffset);
rionWriter.writeInt64(123);
rionWriter.writeFloat64(987.654);
rionWriter.writeUtf8("Rion Ops Example");
int totalBytesWritten = rionWriter.index - startOffset;
This example writes 3 RION fields to the destination byte array. A 64 bit integer, a 64 bit floating point,
and a UTF-8 text field. The last line of the example calculates how many bytes the RionWriter has written
to the destination byte array. You need this information if you want to read the RION fields again with
a RionReader, or if you want to write them to disk, network etc.
The RionWriter index field is updated after every single write of a RION field
to the destination byte array.
Write Bytes
The RionWriter contains a method named writeBytes() which can be used to write
a RION Bytes field to the destination byte array. The writeBytes() takes a byte array as parameter.
The content of this byte array will be written into the RION Bytes field. Here is a Java example of writing a
RION Bytes field from a byte array using the RionWriter writeBytes() method:
rionWriter.writeBytes( new byte[]{1,2,3,4,5} );
The writeBytes() method also exists in a version that takes a source offset and source length
which specifies from what offset and how many bytes forward in the source byte array to write to the
RION Bytes field in the destination byte array. Here is an example that only writes part of the source byte array:
rionWriter.writeBytes( new byte[]{1,2,3,4,5,6}, 2, 3);
This example will write the bytes from offset 2 (value = 3) and 3 bytes forward (until and including value 5) as a RION Bytes field in the destination byte array.
Write Boolean
The RionWriter method writeBoolean() writes a boolean value
(true / false) as a RION Boolean field to the destination byte array. Here is a Java
example of writing a RION Boolean field using the RionWriter writeBoolean() method:
rionWriter.writeBoolean(true);
Write Int64
The RionWriter method writeInt64() will write a 64 bit integer as a RION IntPos or IntNeg
field to the destination byte array. If the int64 value is positive, a RION IntPos field will be written. If the
value is negative, a RION IntNeg field will be written. Here are two examples of writing int64 values
(long values) as RION IntPos and IntNeg fields to the destination byte array, using the
RionWriter writeInt64() method:
rionWriter.writeInt64( 1234567890 ); rionWriter.writeInt64( -1234567890 );
Write Float32 and Float64
The RionWriter methods writeFloat32() and writeFloat64() writes a
32 bit and 64 bit floating point number as a RION Float field to the destination byte array. Here are two
examples of writing a 32 bit and 64 bit floating point number as RION Float fields to the destination byte array,
using the RionWriter writeFloat32() and writeFloat64() methods:
rionWriter.writeFloat32( 123.456F ); rionWriter.writeFloat64( 123456.7890123D );
Write UTF-8
The RionWriter method writeUtf8() writes a Java String
as a RION UTF-8 or UTF-8-Short field to the destination byte array. Whether a UTF-8 or UTF-8-Short field is written
depends on the length of the String. Here is an example of writing a RION UTF-8 field to a byte array using
the RionWriter writeUtf8() method:
rionWriter.writeUtf8("A string to write to UTF-8");
Write UTC
The RionWriter method writeUtc() method writes a date and time in the shape of a
Java Calendar as a RION UTC field (UTC date-time) to
the destination byte array. Here is an example of writing a Java Calendar as a RION UTC field
to the destination byte array using the RionWriter writeUtc() method:
GregorianCalendar calendar = new GregorianCalendar(); rionWriter.writeUtc(calendar, 9);
Notice that the writeUtc() method takes 2 parameters. The first parameter is the Java Calendar
instance with the date-timestamp to write. The second parameter tells the "format" of the RION UTC field.
Here is what these numbers mean in terms of UTC date-time format:
2 = Year 3 = Year + Month 4 = Year + Month + Day 5 = Year + Month + Day + Hour 6 = Year + Month + Day + Hour + Minutes 7 = Year + Month + Day + Hour + Minutes + Seconds 9 = Year + Month + Day + Hour + Minutes + Seconds + Milliseconds
Write Composite RION Fields
RION has three field types that can contain other RION fields nested inside them. These field types are:
- RION Array
- RION Table
- RION Object
Writing composite RION fields is a bit more complex than writing atomic field fields. The reason for that is, that RION fields use a Type-Length-Value (TLV) encoding. The complexity comes from having to write the length of the field already in the beginning of the field. If you don't know at the time of writing the beginning of a RION Object field how many bytes its nested fields will take up, you cannot correctly write the length of the RION Object field at this time.
The solution is to reserve a number of bytes for the length in the beginning of the composite RION field, and when you are done writing all its nested fields, jump back up to these bytes and set the length correctly.
Obviously, in order to be able to reserve enough length bytes for a composite RION field when you start writing it, you must know its maximal possible length. For instance, if you know the field value will never exceed 256 bytes, you only need to reserve 1 byte to represent its length. Similarly, if the length will never exceed 65,536 bytes, you only need to reserve 2 bytes for the length etc.
Using this method there is a risk of reserving more bytes than what the composite RION field ends up needing, but this is usually not a big overhead compared to the total size of the RION field. If the composite RION field can be up to 16MB in length, you need to allocate 3 bytes to represent its length. If the field ends up only needing 1 byte to represents its length, you will have wasted 2 bytes. However, the advantage of having a TLV format when reading the RION field by far outweighs these wasted length bytes.
The RionWriter class has functionality that helps you reserve length bytes for composite RION fields
and jump back up and fill them in correctly later. You will see how that works in the later sections for each of the
composite RION fields.
Setting the Nested Field Stack
One of the mechanisms in the RionWriter class that helps you reserve length bytes and set their value
correctly later requires an internal stack to keep track of where the reserved length bytes are located. This
stack is referred to as a nested field stack. The nested field stack is just an int array.
The length of the nested field stack governs how many levels of nesting the given RionWriter will
be able to handle. You set the nested field stack via the setNestedFieldStack() method.
Here is an example of setting the nested field stack for a RionWriter via its
setNestedFieldStack() method:
rionWriter.setNestedFieldStack(new int[16])
This example sets an int array of length 16 (with 16 cells) as nested field stack. This means,
that the RionWriter can now handle up to 16 levels of nested RION fields.
Write Object
A RION Object field is a composite RION field which can contain nested RION fields. There are no rules for what fields you nest inside a RION Object field. You can structure the nested fields as you need for the given use case. However, there are two commonly used structures:
- Key, Value Structure
- Values Only Structure
Both of these types of RION Object structures are explained in more detail below.
Key, Value Structure
In the key,value structure you nest pairs of RION fields inside the RION Object field. These pairs consists of a RION Key field and the appropriate RION value field for that key. The RION Key field is typically used to represent a field name, or the key used in a Map / dictionary for that value, much in the same way field names are used in JSON. Here is a simple example of what the key, value structure could look like in RION fields:
RION Object
RION Key
RION IntPos
RION Key
RION UTF-8
Rion Key
RION UTC
Each RION Key field is considered the key / field name for the following RION field. In case a RION Key field
is followed by another field, the value for the first of these Key fields should be considered to be null.
However, it is possible for all RION fields to take the value null, so it is possible to include
e.g. a RION UTF-8 field with a null value, rather than omitting it. However, it would be possible
to omit it to save bytes.
Using the key, value structure the order of the key, value pairs it typically less important. The RION Key field tells the semantic meaning of the following RION value field. However, if the order of the key, value pairs is fixed, you can hand-write some faster key, value field pair processing than you can if the order is not fixed.
Values Only Structure
In the values only structure, the RION Object field does not contain any RION Key fields. Only the RION value fields are nested inside the RION Object field. Here is a simple example of that the values only structure could look like in RION fields:
RION Object
RION IntPos
RION UTF-8
RION UTC
Using the values only structure will result in smaller RION Object fields (less bytes used). However, you have no Key field to tell what the semantic meaning is of the value field. All you have is the index of the value field. E.g. you may decide that the first nested field is a customer ID, the second is the customer name, the third is the customer birthday etc.
Write Object Begin and End
When writing a RION Object field using the RION Ops RionWriter, you tell the RionWriter
when the RION Object begins, how many length bytes to reserve, and when the RION Object ends. Here is an
example of marking the beginning and end of a RION Object using the RionWriter :
byte[] rionDestination = new byte[1024];
RionWriter rionWriter = new RionWriter()
.setNestedFieldStack(new int[16])
.setDestination(rionDestination, 0);
int reservedLengthBytes = 2; //max 65,536 bytes long Object.
rionWriter.writeObjectBeginPush(reservedLengthBytes);
// write nested RION fields
rionWriter.writeObjectEndPop();
The writeObjectBeginPush() method writes the beginning of the RION Object field, and
reserves and reserve as many length bytes as you pass to it as parameter (2 in the example above).
The start index of the RION Object field is pushed onto the nested field stack internally for later use.
The writeObjectEndPop() method pops the RION Object field start index
off the nested field stack, calculates the total length of the Object field and writes the total
Object field length into its reserved length bytes.
There are also versions of the writeObjectBegin() and writeObjectEnd()
method that does not use the internal nested field stack. If you want to use these, you will have to
keep track of the index of the Object field length bytes yourself, though. This is harder, and makes
your code more ugly. But its possible if you prefer.
byte[] rionDestination = new byte[1024];
RionWriter rionWriter = new RionWriter()
.setNestedFieldStack(new int[16])
.setDestination(rionDestination, 0);
int objectStartIndex = rionWriter.index;
int reservedLengthBytes = 2; //max 65,536 bytes long Object.
rionWriter.writeObjectBegin(reservedLengthBytes);
// write nested RION fields
int objectLength = rionWriter.index - objectStartIndex;
rionWriter.writeObjectEnd(objectStartIndex, reservedLengthBytes, objectLength);
This avoids the push and pop on the internal nested field stack, but makes your code uglier. Especially if you have multiple layers of nested fields.
Write Key, Value Object
Here is an example of writing a key, value structured RION Object field using the RionWriter:
byte[] rionDestination = new byte[1024];
RionWriter rionWriter = new RionWriter()
.setNestedFieldStack(new int[16])
.setDestination(rionDestination, 0);
int reservedLengthBytes = 2; //max 65,536 bytes long Object.
rionWriter.writeObjectBeginPush(reservedLengthBytes);
rionWriter.writeKeyOrKeyShort("customerId");
rionWriter.writeInt64( 123 ); //customerId
rionWriter.writeKeyOrKeyShort("name");
rionWriter.writeUtf8("Jane Doe"); //customer name
rionWriter.writeKeyOrKeyShort("name");
rionWriter.writeUtc(new GregorianCalendar(), 7); //customer birthdate (today)
rionWriter.writeObjectEndPop();
Write Values Only Object
Here is an example of writing a values only RION Object field using the RionWriter :
byte[] rionDestination = new byte[1024];
RionWriter rionWriter = new RionWriter()
.setNestedFieldStack(new int[16])
.setDestination(rionDestination, 0);
int reservedLengthBytes = 2; //max 65,536 bytes long Object.
rionWriter.writeObjectBeginPush(reservedLengthBytes);
rionWriter.writeInt64( 123 ); //customerId
rionWriter.writeUtf8("Jane Doe"); //customer name
rionWriter.writeUtc(new GregorianCalendar(), 7); //customer birthdate (today)
rionWriter.writeObjectEndPop();
Write Table
The RION Table field type is capable of encoding tabular data, similarly to how a CSV file works. A RION Table field consists of rows and columns. Each row has the same number of column values. The outline of a RION encoded Table field looks like this:
- Table Field
- Row count field
- 1..N Key or Key Short fields
- 1..N RION fields for row 1
- 1..N RION fields for row 2
- 1..N RION fields for row 3
- ...
The way to specify how many columns each row has is via the first Key or Key Short fields written in the beginning of the Table field. If you write 3 Key or Key Short fields, each row will have 3 columns. These Key or Key Short fields are similar to the column names in the header row of a CSV file.
There is no explicit delimiter between the last Key / Key Short field and the first field of the first row. The first non Key / Key Short field found after the Key / Key short field sequence is considered the first field of the first row.
There is also not any explicit delimiter between the rows. The way you find out when one row ends and another starts is simply by counting the fields. You know from the Key / Key Short sequence how many fields there should be per row.
Here is an example of writing a RION Table with 3 columns and 3 rows.
RionWriter rionWriter = new RionWriter();
rionWriter.setDestination(new byte[1024], 0);
rionWriter.setNestedFieldStack(new int[16]);
// begin table - reserve 2 lengthLength bytes
rionWriter.writeTableBeginPush(2);
// write the 3 column key fields (header fields)
rionWriter.writeKeyShort("col1");
rionWriter.writeKeyShort("col2");
rionWriter.writeKeyShort("col3");
// write row 1, field 1,2,3
rionWriter.writeInt64(1);
rionWriter.writeUtf8("val 1.2");
rionWriter.writeUtf8("val 1.3");
// write row 2, field 1,2,3
rionWriter.writeInt64(2);
rionWriter.writeUtf8("val 2.2");
rionWriter.writeUtf8("val 2.3");
// write row 3, field 1,2,3
rionWriter.writeInt64(3);
rionWriter.writeUtf8("val 3.2");
rionWriter.writeUtf8("val 3.3");
// end table - 3 rows total
rionWriter.writeTableEndPop(3);
| Tweet | |
Jakob Jenkov | |
Sponsored Ads
