RIAPS Platform Information

Logo

RIAPS is a Resilient Middleware for Smart Grids. See riaps.isis.vanderbilt.edu for more information.

View My GitHub Profile

Component Logging Tutorial

The RIAPS framework utilizes spdlog (a fast C++ logging library) to provide a logging mechanism for both the RIAPS framework and the application components. There are 3 ways to log information within a component: default, actor-level, and customizable component-level logging.

Page Contents

Default Logging

Use of logging within a component is simple. Utilize the RIAPS component self.logger.<log level> function to provide user information. Here is an example:

self.logger.info("on_tempupdate(): Temperature:%s, PID %s, Timestamp:%s" % (temperatureValue, str(now), temperatureTime))

The available log levels are listed below in order of priority (from low to high). All logging higher than the level set will be provided (i.e. for warn level setting, the error and critical messages will also be provided). The RIAPS platform sets a log level of info by default. At this level, you will receive all messages at the info level and others of higher priority (such as warn, error and critical).

For initial debugging efforts, the component logging level can be adjusted using the following statements in the application component code.

import spdlog as spd

logger.set_level(spd.LogLevel.INFO)

Note: Once component code works and matures, it is preferred to set the level using the Customizable Component-Level Logging described below so that tested code does not have to change.

The log messages are sent to the console output on the node. If working on the development VM and starting the riaps_deplo using either Eclipse or sudo -E riaps_deplo in a terminal window, the console output will be available. The information provided will be both the RIAPS platform and the application component logging information.

The target nodes nodes run riaps_deplo as a systemd service, so the console log information is placed in the system logs. To view the information as the application is running, ssh into a target node and run the journalctl command or you can view an individual target node journal file using a fabric command.

Actor-Level Logging

A global way to log Actor information to a file in the deployed application directory is to turn on the RIAPS Framework application log option in the RIAPS Framework Configuration file. This is done by adding the ‘log’ designations to the ‘app_logs =’ configuration line, as shown below. This file is located on each RIAPS node in /usr/local/riaps/etc/riaps.conf. Each actor defined in the application model file will have a log file in /home/riaps/riaps_apps/<app name>/.

# application logs
# ''        -- stdout
# log       -- app/actor.log file
app_logs = log

For the remote RIAPS nodes, the riaps.conf can be modified on the development VM and then moved to the remote nodes using the fabfile utility. Start with original riaps.conf file and configure appropriately for your system configuration (including the NIC name). Make sure the RIAPS nodes are accessible from the fab command (see riaps_fab sys.check). Then use the riaps_fab riaps.updateConfig command to transfer the locally edited riaps.conf file to the remote nodes.

Since applications are deployed under a generated username, the developer must have root access to view the log messages in the /home/riaps/riaps_apps/<app name> directory. When writing to a log file, output to this file happens in batches so it can take some time for the data to appear in the desired file (on the order of a minute or more).

The /home/riaps/riaps_apps/<app name> directory is only available when the application is deployed. Removing an application using the RIAPS controller will delete the log information. It is anticipated that when an application is deployed in a real system, it could be stopped and restarted or just left continuously running throughout the life of the application on the system. Therefore, logged information will remain available to system operators. During application development, the developer will be deploying and removing the application during debugging efforts. So it is recommended to copy the logs to another location if access is needed at a later time (for debugging). The log files can be copied from a remote node to the development machine using the following fabric utility command from the development machine. All available application log files (.log) will be retrieved from each node for the application identified (app_name). For each node, these files will be placed under a logs/ directory..

riaps.getAppLogs -H riaps-8014.local

Note: It is not recommended to leave actor-level logging on in a real system deployed application unless the logging is for errors and warnings only, in order to minimize disk usage. Consider using customizable component-level logging instead.

Customizable Component-Level Logging

The application developer can customize the logging setup for each application component instance by defining a riaps-log.conf file and placing it in the application directory with the components. This file defines the logging configuration and output pattern for each of the application components instances, thus providing flexibility for the developer to direct and format the output of each component instance individually.

The logging configuration (‘[[logger]]’) indicates the component instance for the specific configuration definition, how the output will be handles, and the output format used. When the actors were defined in the application model file (.riaps), a name was given for each instance of the application component used in the actor. Using the WeatherMonitor example application, an actor definition is shown below. The ‘name’ for this component instance will be ‘WeatherIndicator.sensor’, where ‘WeatherIndicator’ is the actor’s name and ‘sensor’ is the component instance name.

    actor WeatherIndicator {
       {  // TempSensor publishes 'TempData' messages
          sensor : TempSensor;
       }
    }

Device components are handled slightly differently in the RIAPS platform. The device component instance is identified by the device component name followed by the same name. Using the DistributedEstimatorGPIO example application, an actor definition (shown below) would be called by a name of GPIODevice.GPIODevice.

    device GPIODevice() {
        sub blink : Blink;
    }

    // Estimator actor
    actor Estimator (freqArg,value=0.0) {
        local SensorReady, SensorQuery, SensorValue, Blink;    // Local message types
        {  // Sensor component
            sensor : Sensor(value=value);								
            // Local estimator, publishes global message 'Estimate'
            filter : LocalEstimator(frqArg=freqArg);
            // GPIO component to blink
            gpio : GPIODevice();
        }
    }

In spdlog terms, a sink (‘[[sink]]’) is an interface between a logger instance and its desired target. Each logger is associated with one or more sink objects that actually write the log contents to a specific target such as a file, console or database. Each logger instance also contains its own private instance of a formatter object, which is responsible for the visual representation of the logged data. It can be customized with user-defined patterns (‘[[pattern]]’) that specify the log format.

Console and file sink types available are:

Note: Utilize either single threaded (‘_st’) or thread safe multi-threaded (‘_mt’) loggers. While single threaded sinks cannot be used from multiple threads simultaneously, they are faster since no locking is employed. For a full list of loggers, see Supported Sinks

For the rotating log file, when the maximum size is reached, a new file is created and the logger switches to a new file. The maximum size of each file and the maximum number of files can be configured.

The pattern section defines the expected format of the logging message, any identifying text, and the message logged by the component in the code. The pattern flag options are listed in a spdlog custom formatting tutorial.

Using the example WeatherMonitor application, an example riaps-log.conf is as follows, showing custom patterns for each component being sent to the console for display:

#
# Log configuration example
#

[[sink]]
name = "console_mt"
type = "stdout_sink_mt"

# Override pattern for WeatherIndicator.sensor
[[pattern]]
name = "sensor_console"
value = "[%l]:%H:%M:%S,%e:[%P]:SENSOR:%v"

[[logger]]
name = "WeatherIndicator.sensor"
sinks = ["console_mt"]
pattern = "sensor_console"

# Override pattern for WeatherReceiver.monitor
[[pattern]]
name = "monitor_console"
value = "[%l]:%H:%M:%S,%e:[%P]:MONITOR:%v"

[[logger]]
name = "WeatherReceiver.monitor"
sinks = ["console_mt"]
pattern = "monitor_console"

To configure a file sink, include the log filename and logging level. If creating a directory for the logs, add the ‘create_parent_dir’ directive.

[[sink]]
name = "weather_indicator_file_mt"
type = "basic_file_sink_mt"
filename = "log/windicator_file.log"
level = "info"
create_parent_dir = true

Note: When writing to a log file, output to this file happens in batches so it can take some time for the data to appear in the desired file (on the order of a minute or more). It is highly recommended to write each component instance to a separate file. If multiple components are writing to the same file, each batch output will write logs for a single component rotating until all components output a batch set (not interleaved based on time). During the batch transitions, the component output can become mixed together with another component output. While all data remains present, batch transition lines become hard to parse for desired information.

Example configuration files for various sink types can be found at under TOML Configuration Example. If a log folder is desired for daily and rotating log files, a basic file sink definition will be needed to create this folder using the ‘create_parent_dir’ indicator since application folders are dynamically created when the RIAPS application is deployed (not by a system administrator). The basic file log will be created in the desired log directory, but will not be used if it is not specified by a component logger definition.

The console information output can be viewed as indicated in the Default Logging section. Like the actor-level logging, the log files will be stored in the deployed application location (/home/riaps/riaps_apps/<app name>/). To access the files, the developer must have root access.

The /home/riaps/riaps_apps/<app name> directory is only available when the application is deployed. Removing an application using the RIAPS controller will delete the log information. It is anticipated that when an application is deployed in a real system, it could be stopped and restarted or just left continuously running throughout the life of the application on the system. Therefore, logged information will remain available to system operators. During application development, the developer will be deploying and removing the application during debugging efforts. So it is recommended to copy the logs to another location if access is needed at a later time (for debugging).

Each application can specify resource limits on each actor instance, one of which is disk usage (or file space). This limit is specified in the application model file, see the Resource Management Specifications section. All components that create log files will be managed by their respective actor-level file space limitation, where the specified limit applies to the combined file space use of all the components that make up the specific actor instance.

RIAPS Framework Logging

By default, minimal logging is configured to indicate the current system status and any errors encountered by the RIAPS framework. More logging is available for each of the RIAPS framework modules to assist in debugging issues. The logging configuration file for the RIAPS framework can be found on each of the RIAPS nodes in usr/local/riaps/etc/riaps-log.conf Additional module information can be added by including the desired module (such as ‘riaps.deplo.depm’) in the comma separated ‘keys’ list. The default framework riaps-log.conf file is available in the RIAPS Github repository.