Raw Export Collector Plugin

Introduction

The Raw Export plugin implments a new storage type for Torrus. It allows exporting the raw collected data in plain files. Currently the counter values are exported without changes, and in the future there will be support for exporting counter increments.

The data is exported in text form, one datasource per line. Each line contains three or four fields, separated by the characters defined by raw-field-separator parameter:

The raw export files are written by a background thread, leaving the main thread more time to perform the collecting work. The torrus schedulerinfio reports the size of the raw update queue in runtime statistics.

Installation

Follow the Torrus installation guide for the main software installation. This plugin requires the multithreading support in Torrus. It would not function in a non-multithreaded environment.

Unpack the plugin distribution package in some directory:

  gzip -dc tp-rawexport-1.X.tar.gz | tar xvf -

Then run the Torrus plugin installation utility:

  torrus install_plugin tp-rawexport-1.X

Restart Torrus collector processes.

Configuration

In order to get use of Raw export, the storage-type parameter should contain raw as one of the comma-separated values. Usually storage-type is set to the value raw,rrd.

The following parameters define the raw storage operation:

Example

The following XML template is installed as examples/rawexport-example.xml by the plugin installer. It provides basic functionality for exporting of IF-MIB counters. The example refers to a parameter raw-file-suffix which is not part of this plugin, and is given only as an example. The discovery instruction (DDX) file listed below defines this parameter at the host level.

Note that your templates should also declare the parameters as expandable, as given in the eample below.

It is important that raw-file depends on collector-instance. It ensures that several collector instances for a given tree would never write to the same file concurrently.

This example stores the data in files that depend on the calendar date. This makes them convenient to grab for daily processing. For example, at 3:00 every morning you can move all the files that are more than 2 hours old for further processing. Also their alphabetical sorting would list them in temporal order.

File: examples/rawexport-example.xml

 <configuration>
   <param-properties>
     <prop param="raw-file"       prop="expand" value="1"/>
     <prop param="raw-rowid"      prop="expand" value="1"/>
   </param-properties>
   <datasources>
     <template name="raw-defaults-ifoctets-examle">
  
       <param name="raw-datadir" value="/srv/torrus/raw" />
       <param name="raw-file"
              value="#Y-#m-#d.%collector-instance%.%raw-file-suffix%" />
       <param name="raw-field-separator" value="," />
       <param name="raw-timestamp-format" value="#d#b#Y:#H:#M:#S" />
       <param name="raw-rowid" value="%system-id%,%interface-name%,%rrd-ds%" />
    
     </template>
  
   </datasources>
 </configuration>

The following is a fragment of a Devdiscover instructions file (DDX) which sets up the raw export for input and output byte counters, for all interfaces whose names start with 'eth':

  <host>
    <param name="snmp-host" value="mylinuxhost"/>
    <param name="include-files" value="examples/rawexport-example.xml"/>
    <param name="custom-host-templates" value="raw-defaults-ifoctets-examle"/>
    <param name="host-copy-params" value="raw-file-suffix"/>
    <param name="raw-file-suffix" value="rawexport"/>
    <param name="selectors"            value="raw"/>
    <param name="raw-selector-type"    value="RFC2863_IF_MIB"/>
    <param name="raw-selector-expr"    value="{ifSubtreeName}"/>
    <param name="raw-ifSubtreeName"    value="^eth"/>
    <param name="raw-selector-actions"
                             value="InBytesParameters,OutBytesParameters"/>
    <param name="raw-InBytesParameters-arg">
      storage-type=raw,rrd;
      raw-counter-base=64;
      raw-counter-maxrate=1e15
    </param>
    <param name="raw-OutBytesParameters-arg">
      storage-type=raw,rrd;
      raw-counter-base=64;
      raw-counter-maxrate=1e15
    </param>
  </host>