Tech and Media Labs
This site uses cookies to improve the user experience.




IonReader

Jakob Jenkov
Last update: 2017-03-08

The IonReader class (com.nanosai.gridops.ion.read.IonReader) is capable of iterating through binary ION data stored in a byte array. The IonReader does not have a static set of APIs. Only instance methods. This tutorial will go through how to use the IonReader to read both primitive as well as complex ION fields.

Creating an IonReader

Before you can use the IonReader you must create an IonReader instance and attach it to a source byte array. Here is an example showing how to create an IonReader:

byte[] src = new byte[10 * 1024];
int ionDataLength = 0;

// write ION data into src byte array
// and set ionDataLength to length of ION data.

IonReader reader = new IonReader(src, 0, ionDataLength);

You can also create an IonReader and attach it to a byte array later. Here is how that looks:

IonReader reader = new IonReader();

byte[] src = new byte[10 * 1024];
int ionDataLength = 0;

// write ION data into src byte array
// and set ionDataLength to length of ION data.

reader.setSource(src, 0, ionDataLength);

Iterating the ION Fields

Once you have created an IonReader you can iterate through the ION fields in the attached byte array. The IonReader has a method called hasNext() which returns true as long as there is more data to read in the attached byte array. Thus you can iterate the ION fields in the byte array like this:

IonReader reader = new IonReader();

// attach IonReader to a byte array with ION fields here...
reader.setSource(src, 0, ionDataLength);

while(reader.hasNext()) {
    reader.next();
    reader.parse();

    //read the value of each ION field.

}

As you can see, iterating with an IonReader is pretty similar to iterating using a Java Iterator.

The hasNext() method looks the ionDataLength provided in the IonReader constructor or in setSource() to determine if there are more ION fields in the byte array. Therefore it is important that the length of the ION field data (in bytes) is provided correctly.

The next() method moves the IonReader to the next ION field if there is any. Actually, it just moves the internal read pointer to the first byte after the current ION field.

Calling next() just after having called setSource() will move the internal pointer to the first ION field in the byte array. Before the first next() call the internal pointer points "before" the first ION field.

The parse() method parses the ION field the IonReader is currently pointing to in the source byte array. When the field is parsed the field type, lenght-length and length are all stored in public variables of the IonReader, which you can read to process the ION field.

Exactly what you want to do inside the while loop depends on what type of ION data you have in the byte array. If you have a sequence of log entries, then you iterate through each field and process them separately. If you have an ION Object, you might need to parse into that object and read its nested fields, objects etc. You will see how that is done later in this tutorial.

Public Variables

The IonReader has a set of public variables which you can use to inspect or process the ION fields during iteration. The variables are public so that you can easily access them, but you should be really careful about writing to them. They are primarily meant for reading - without the need for getter methods.

The public variables are set when you call the various methods of the IonReader. The public variables are:

VariableDescription
source The source byte array
index Then index of the body of the current field (the body, not the lead byte).
nextIndex The index of the lead byte of the next field.
fieldType The ION field type of the current field. The value will match one of the constants in IonFieldTypes
fieldLengthLength The length-length of the current field (the number of bytes used to represent the length of the field). If the current field uses a Short encoding, the fieldLengthLength and fieldLength will have the same value.
fieldLength The length in bytes of the body (value) of the current field.

Reading Primitive ION Fields

Reading primitive ION fields with the IonReader is pretty simple. When the IonReader is positioned at a primitive ION field, call the corresponding primitive ION field reader method. The primitive ION field read methods are covered in the following sections.

readBoolean()

The readBoolean() methods reads the value of the current ION field into a boolean value. The ION field must be an ION Tiny field with the value of 1 (true) or 2 (false) (0 means null). Here is an IonReader readBoolean() example:

boolean value = reader.readBoolean();

readInt64()

The readInt64() method reads the value of the current ION field into a long value. The ION field must be an ION Int64-Positive> or Int64-Negative field. Here is a IonReader readInt64() example:

long value = reader.readInt64();

readFloat32()

The readFloat32() method reads the value of the current ION field into a Java float value. The ION field must be an ION Float with the length 4 (meaning a length of 4 bytes). Here is an IonReader readFloat32() example:

float value = reader.readFloat32();

readFloat64()

The readFloat64() method reads the value of the current ION field into a Java double value. The ION field must be an ION Float with the length 8 (meaning a length of 8 bytes). Here is an IonReader readFloat64() example:

double value = reader.readFloat64();

readUtf8String()

The readUtf8String() methods reads the value of the current ION field into a Java String. The ION field must be an ION UTF-8 or UTF-8-Short field. Here is an IonReader readUtf8String() example:

String value = reader.readUtf8String();

readUtcCalendar()

The readUtcCalendar() method reads the value of the current ION field into a Java Calendar object. The ION field must be an ION UTC field. Here is an IonReader readUtcCalendar() example:

Calendar value = reader.readUtcCalendar();

Arbitrary Hierarchical Navigation

Reading complex ION fields like Object, Table and Array fields is a bit more complicated than just reading primitive type ION fields. However, once you understand how to control this arbitrary hierarchical navigation, you will see that it is a quite powerful way to navigate ION data. It gives you full control over how deep into the hierarchy you want to navigate, and makes it easy and fast to skip over the parts of the hiearchy you are not interested in.

When you iterate through the ION data in the source byte array you will normally iterate from one ION Object, Table or Array directly to the next ION field at the same level (the next "sibling" of the previous ION field, so to speak).

In order to read the ION fields nested inside an ION Object, Table or Array field you must instruct the IonReader to parse into the complex ION field. You do so by calling parseInto().

Once you have called parseInto() the IonReader will iterate the ION fields nested inside that ION field. Thus, the IonReader hasNext() method changes its semantics to tell whether or not the ION field has more nested elements or not. Thus, you can still iterate the nested fields of a complex ION field using a while(reader.hasNext()) loop.

Once you have reached the end of a complex ION field (Object, Table or Array) you need to call parseOutOf() to step out of the complex ION field. Calling parseOutOf() returns the IonReader to the ION field you had just parsed into. Calling parseInto() would thus jump into the same ION field again. To get to the next ION field, call next() .

Here is a simplified arbitrary hierarchical navigation example:

while(reader.hasNext()) {
    reader.parse();

    if(reader.fieldType == IonFieldTypes.OBJECT) {
        reader.moveInto();

        while(reader.hasNext()) {
            reader.next();
            reader.parse();

            //read nested ION field

        }

        reader.moveOutOf();
    }

    reader.next();
}

This example shows a navigation of two hierarchy levels. There is the outer level which consists of a stream of ION Object fields. And, there is an inner level which consists of the fields nested inside these ION Object fields.

In reality the level of nesting could be higher than just 2. If, for instance, the inner nested part of the above code (the part that reads an ION Object),

Reading Objects

Reading ION Objects using the IonReader requires more code than reading primitive fields. ION Objects typically consists of ION Key field + another ION field pairs (although this is not strictly required). First you have to read the Key field, and then you need to determine what field in the Java object you are reading to set. Here is an example of reading an ION Object using the IonReader:

while(reader.hasNext()) {
    reader.parse();

    List<Employee> employees = new ArrayList<>();

    if(reader.fieldType == IonFieldTypes.OBJECT) {
        reader.moveInto();

        Employee employee = new Employee();

        while(reader.hasNext()) {
            reader.next();
            reader.parse();

            // Read property key (name)

            String key = null;
            if(reader.fieldType == IonFieldTypes.KEY) {
                key = reader.reader.readKeyAsUtf8String();
            } else if(reader.fieldType == IonFieldTypes.KEY_SHORT) {
                key = reader.readKeyShortAsUtf8String()
            }

            // Read property value

            reader.next();
            reader.parse();

            if("firstName".equals(key)){
                employee.setFirstName(reader.readUtf8String());
            } else if("lastName".equals(key)){
                employee.setLastName(reader.readUtf8String());
            } else if("employeeId".equals(key)) {
                employee.setEmployeeId(reader.readInt64());
            }
        }

        employees.add(employee);

        reader.moveOutOf();
    }

    reader.next();
}

As you can see, the matching of property names (ION Key fields) to the corresponding properties in a Java object currently requires a sequence of if - else if statements. This works okay for objects with fewer properties, but when the number of properties rise, going through this list of if - else if statements gets slower and slower because there are more if statements to test.

We are currently working on ways to improve both the readability and speed of reading objects using the IonReader. There are at least 2 other ways to do it which are both faster. We will update this tutorial when these methods are ready for use.

Reading Tables

Reading ION Tables using the IonReader also takes some code, but it is possible too. Remember, ION Tables start with a series of ION Key fields followed by a series of ION fields that are associated with the Key fields based on their index. Here is an example of reading ION Tables using the IonReader:

while(reader.hasNext()) {
    reader.parse();


    if(reader.fieldType == IonFieldTypes.TABLE) {
        reader.moveInto();
        

        List<Employee> employees = new ArrayList<>();

        // First, parse keys into a list of String

        List<String> keys = new ArrayList<>();

        reader.next();
        reader.parse();
        while(reader.fieldType == IonFieldTypes.KEY ||
              reader.fieldType == IonFieldTypes.KEY_SHORT) {

            keys.add(reader.readKeyAsUtf8String());

            if(reader.hasNext()) {
                reader.next();
                reader.parse();
            }
        }

    
        //Second, read the field values and turn them into Java objects
        int fieldIndex = 0;
        Employee employee = null;

        while(reader.hasNext()) {

            if(fieldIndex == 0) {
                employee = new Employee();
                employees.add(employee);
            }

            String key = keys.get(fieldIndex);

            if("firstName".equals(key)){
                employee.setFirstName(reader.readUtf8String());
            } else if("lastName".equals(key)){
                employee.setLastName(reader.readUtf8String());
            } else if("employeeId".equals(key)) {
                employee.setEmployeeId(reader.readInt64());
            }

            fieldIndex++;
            if(fieldIndex == keys.size()) {
                fieldIndex = 0;
            }

            if(reader.hasNext()) {
                reader.next();
                reader.parse();
            }

        }

        reader.moveOutOf();
    }

    reader.next();
}

This code is longer than for reading an ION Object. However, once the code starts parsing ION values fields (after having parsed the ION Key fields at the beginning of the ION Table), Key fields are looked up based on the index of value field being read. This is faster than having to parse a Key field for each property of each object (row) in the Table.

Reading Arrays

You can also read ION Array fields with the IonReader. Reading ION Arrays is simpler than reading Objects and Tables. Here is an example of reading ION Arrays using the IonReader:

while(reader.hasNext()) {
    reader.parse();

    if(reader.fieldType == IonFieldTypes.ARRAY) {
        reader.moveInto();
        reader.next();
        reader.parse();

        //first read element count
        int elementCount = reader.readInt64();

        String[] elements = new String[elementCount];

        int index = 0;
        while(reader.hasNext()) {
            reader.next();
            reader.parse();

            elements[index] = reader.readUtf8AsString();
        }

        reader.moveOutOf();
    }

    reader.next();
}

Jakob Jenkov




Copyright  Jenkov Aps
Close TOC