The Custom feed type allows the raw document to be processed by JavaScript to construct an appropriate tabular dataset.

Script References

This is an opportunity to explicitly declare any references the script makes to resources on the filesystem or otherwise, which would be required to be included in any partial backups made where this JavaScript feed is involved.

Script Implementation

Users are required to implement two functions: getAttributes(inputString, nodeVars, secVars) and getRecords(inputString, nodeVars, secVars).

Arguments

• inputString: A string containing the raw data to be processed.
• nodeVars: An object whose properties correspond to any variables defined for this feed. To access a variable, use nodeVars.varName.
• secVars: An object whose properties correspond to any secured variables defined for the current user. To access a variable, use secVars.varName.

getAttributes(inputString, nodeVars, secVars)

This function must return one of the following:

• jsAttributesSuccess(attributes, “Optional success message for debugging”)
• jsAttributesFailure(“Optional failure message”)

In the success case, attributes are an array of objects, where each object represents an attribute definition. Attribute definition objects have the following properties:

• name: required, should consist of alphanumerics and underscores only with no leading, trailing, or adjacent underscores.
• type: valid types are ‘string’, ‘int’, ‘long’, ‘number’, ‘boolean’, and ‘date’. If not set, ‘string’ is used as the default type.
• isId: true or false. If not set, false is the default value.
• units: valid units are ‘millis’ or ‘seconds’. This property must be set if the type is ‘date’ and the data values consist of Unix timestamps in milliseconds or seconds, respectively.
• format: valid formats consist of pattern strings such as those documented in the SimpleDateFormat Javadoc. This property should be set if the type is ‘date’ and the data values consist of parseable date strings. If not set, the server will attempt to identify the format, but the output may not be as predictable as if the user were to define the format ahead of time.

getRecords(inputString, nodeVars, secVars)

This function must return one of the following:

• jsRecordsSuccess(records, “Optional success message for debugging”)
• jsRecordsFailure(“Optional failure message”)

In the success case, records is an array of objects, where each object represents a record. Record property names should match attribute names defined in getAttributes(inputString, nodeVars, secVars). For example, if a string attribute “foo” is one of the defined attributes, one can assign var record =  {‘foo’ : bar}.

Additional Services

Some situations require fetching the results of other data producers within the pipeline. This can be achieved via the dataProducerService object.

dataProducerService API for edgeSuite 3.7.x and newer

fetchRecords(producerName, nodeVars, secVars)

This function takes the name of the producer, as well as the nodeVars and secVars objects. Generally speaking, the nodeVars and secVars objects can be passed from the parent function (see above documentation for details). However, nodeVars can also be defined as an object with key:value pairs, which are then internally converted as required by the server.

The returned value is an array of objects (records) whose properties correspond to the attribute names of the resulting data set. The value of a record for a given attribute can then be accessed via record.attrName, e.g. record.foo = “bar”.

dataProducerService API for edgeSuite 3.6.x and older

fetch(producerName, nodeVars, secVars)

This function behaves similarly to the one above, the main difference being that the returned value is a Java object of type DataResultsDO. Use of this function is no longer recommended, however versions as old as 3.4.x can leverage a simple workaround to achieve functionality similar to that of 3.7.x. Assuming the returned DataResultsDO object is defined, and its data is both defined and of type List<TabularRecordDO>, users may convert the data to an array of JavaScript objects by calling convertSourceRecordsForJs([returnedObject.getData()])[0]. This conversion call will generate an array of objects similar to the result of fetchRecords(producerName, nodeVars, secVars) in 3.7.x.

In addition to getData(), the returned DataResultsDO object has a method getResultStatusCode(), which returns an enum, the value of which is one of CURRENT, ERROR_STALE, ERROR_MISSING, REQUIRE_CREDENTIALS, or BAD_CREDENTIALS.

Log Information

Basic logger capability is provided to all runtime scripts via:

yellowstone/server/src/main/java/edge/server/script/ScriptEngineService.java

To get information into a log file, use the following syntax:

if ( logger.isFatalEnabled() ) {
    logger.fatal("Failed to calculate status level ...");
}
if ( logger.isErrorEnabled() ) {
    logger.error("Failed to calculate status level ...");
}
if ( logger.isWarnEnabled() ) {
    logger.warn("Warning: calculation of status level ...");
}
if ( logger.isInfoEnabled() ) {
    logger.info("calculation of status level: {}, completed in: {} seconds", level, instant);
}
if ( logger.isDebugEnabled() ) {
    logger.debug("calculation of status level: {}, completed in: {} seconds", level, instant);
}
if ( logger.isTraceEnabled() ) {
    logger.trace("calculation of status level: {}, completed in: {} seconds", level, instant);
}

More information on the Log4j API can be found at the link below:

https://logging.apache.org/log4j/2.0/log4j-api/apidocs/org/apache/logging/log4j/Logger.html

Example: HTML Parsing

This page provides a working example of parsing HTML weather data using a Custom JavaScript feed to create data that can be consumed in Edge Visualizations.

Prerequisites

This example relies on a java library called jsoup: https://jsoup.org/download. The core library jar was installed in the <edge-home>/lib. The example is known to work with jsoup-1.10.3.jar.

URL for web data source: http://www.weather.gov/data/obhistory/KPWA.html.

Perform the following steps to complete this example:

1. Download and save the jsoup JAR file (see above) into the directory: <edge-home>/lib
2. Create a new Web Data connection with Destination = http://w1.weather.gov/
3. Create a Feed off of the Web Data Connection of “Custom” feed type:
   – Script: enter the script shown below
   – Start URI: /data/obhistory/KPWA.html
4. Go to Data Preview step to view the parsed dataset.

var jsoup = Java.type('org.jsoup.Jsoup');
load('classpath:static/lib/moment-with-locales.min.js');
 
function getAttributes(inputString, nodeVars, secVars) {
    var attributes = [];
    attributes.push({ name : 't', type : 'date', isId : true , units : 'millis' });
    attributes.push({ name : 'Wind' });
    attributes.push({ name : 'Visibility', type : 'number' });
    attributes.push({ name : 'Weather' });
    attributes.push({ name : 'Sky' });
    attributes.push({ name : 'Temperature', type : 'int' });
    attributes.push({ name : 'Dewpoint', type : 'int' });
    attributes.push({ name : 'Humidity', type : 'int' });
    attributes.push({ name : 'Rain', type : 'number' });
    return jsAttributesSuccess(attributes, "Example Success");
}
 
function getRecords(inputString, nodeVars, secVars) {
    var records = [];
    var doc = jsoup.parse(inputString);
    var rows = doc.select('table[cellspacing=3] tr[bgcolor="#eeeeee"],table[cellspacing=3] tr[bgcolor="#f5f5f5"]');
    rows.forEach(function(row) {
        var tds = row.select('td');
        var day = tds.get(0).text();
        var time = tds.get(1).text().split(':');
        var t = moment().startOf('day');
        while (t.date() != day) {
          t = t.subtract(1, 'days');
        }
        t.hour(time[0]);
        t.minute(time[1]);
        var txt = tds.get(15).text();
        var rain = (txt.length > 0) ? parseFloat(txt) : 0;
        // Create a new record and set the attributes defined in getAttributes
        var record = {
            't' : t.unix()*1000,
            'Wind' : tds.get(2).text(),
            'Visibility' : tds.get(3).text(),
            'Weather' : tds.get(4).text(),
            'Sky' : tds.get(5).text(),
            'Temperature' : parseInt(tds.get(6).text()),
            'Dewpoint' : parseInt(tds.get(7).text()),
            'Humidity' : parseInt(tds.get(10).text()),
            'Rain' : rain
        };
        records.push(record);
        // Or enrich a record from the source records
    });
    return jsRecordsSuccess(records, "Example Success");
}

Example 2

function getAttributes(sourceAttrs, nodeVars, secVars, sourceRecords) {
  var attributes = [];
  
  /*
  attributes.push({
  	name: 'octet_in',
    type: 'number',
    isId: false
  });
  attributes.push({
  	name: 'octet_out',
    type: 'number',
    isId: false
  });
  attributes.push({
  	name: 'if_speed',
    type: 'number',
    isId: false
  });
  attributes.push({
  	name: 'timestamp',
    type: 'number',
    units: 'seconds',
    isId: false
  });
  */
  attributes.push({
  	name: 'result',
    type: 'string',
    isId: false
  });
  
  /*
  if (attributes && attributes.length > 0) {
  	logger.info(JSON.stringify(attributes));
  }
  */
  
  return jsAttributesSuccess(attributes, "Example Success");
}

function getRecords(sourceRecords, nodeVars, secVars, attributes) {
  function subtractValuesFromArrays(arr1, arr2) {
    arr1.map(function(num, idx) {
      return num - arr2[idx];
    });
  }
  
  //logger.info(JSON.stringify(sourceRecords));
  var elements = [];
  var deltaTimeInSeconds = 793;
  var deltaIfInOctets;
  
  var octetOut = sourceRecords['d_octet_out'];
  var octetIn = sourceRecords['d_octets_in'];
  var speed = sourceRecords['Speed'][0].ifSpeed;
  
  //logger.info("Octet out: " + JSON.stringify(octetOut));
  //logger.info("Octet in: " + JSON.stringify(octetIn));
  logger.info("speed: " + speed);
  
  octetOut.forEach(function(item, index) {
    elements.push(item.value);
  });
  
  var filteredPairs = elements.filter(function(_, i) { 
    return i % 2 === 0;
  });
  var filteredOdds = elements.filter(function(_, i) { 
    return i % 2 !== 0;
  });
  
  var result = filteredPairs.map(function(num, idx) {
  	return num - filteredOdds[idx];
  });
  
  // logger.info("RESULTS:" + JSON.stringify(result));
  var records = [];
  var record;
  result.forEach(function(el, index) {
  	//deltaIfInOctets = el * 8 * 100;
    // assign outputUtilization result.
    // Create a new record and set the attributes defined in getAttributes
    record = { 
      'result': (el * 8 * 100) / (deltaTimeInSeconds * speed)
    };
    records.push(record);

    // Or enrich a record from the source records
    octetIn[index].result = (el * 8 * 100) / (deltaTimeInSeconds * speed);
    records.push(octetIn[index]);
  });
  
  return jsRecordsSuccess(records, "Example Success");
  // can optionally return jsRecordsFailure(failureMsg) to handle failures
}

• Known issue in 4.4.1 and 4.4.2:
  Due to the GraalVM update, load('classpath:static/lib/moment-with-locales.min.js'); does not work.
• As of 4.4.2-hotfix3, MomentJS is automatically loaded into the context: