The OneFS ilog service is a general logging facility for the cluster, allowing applications and services to rapidly decide if or where to log messages, based on the currently active logging configuration. Historically, OneFS used syslog directly or via custom wrappers, and the isi_ilog daemon provides features common to those wrappers plus an array of other capabilities. These include runtime-modification, the ability to log to file, syslog, and or stderr, additional context including message plus ‘component’, ‘job’, and ‘thread_id’, and default fall-back to syslog.
Under the hood, there are actually two different ilog components; kernel ilog and userspace ilog.
Kernel ilog controls log verbosity at runtime, avoids installing a new kernel module to enable more log detail, and allows only enabling such detailed logging at certain times. Ilog defines six logging levels: Error, Warning, Notice, Info, Debug, and Trace, with levels ‘error’, ‘warning’ and ‘notice’ being written to /var/log/messages with the default configuration. The user interface to kernel Ilog is through sysctl variables, each of which can be set to any combination of the logging levels.
Userspace ilog, while conceptually similar to the kernel implementation, lacks single memory space and per-boot permanence of sysctl variables. User-space processes may start and terminate arbitrarily, and there may also be multiple processes running for a given service or app. Consequently, user-space ilog uses a gconfig file and shared memory to implement run-time changes to logging levels.
Runtime control of OneFS services’ logging is via the ‘isi_ilog’ CLI tool, which enables:
- Adjusting logging levels
- Defining tags which enable log lines with matching tags
- Logging by file or file and line number
- Adding or disabling logging to a file
- Enabling or disabling logging to syslog
- Throttling of logging so repeated messages aren’t emitted more than N seconds apart.
For userspace log, when an application or service using ilog starts up, its logging settings are loaded from the ilog gconfig tree, and a small chunk of shared memory is opened and logically linked to that config. When ilog’s logging configuration is modified via the CLI, the gconfig tree is updated and a counter in the shared memory incremented.
The OneFS applications and services that are currently integrated with ilog include:
| Service | Daemons |
| API | PAPI, isi_rsapi_d |
| Audit | isi_audit_d, isi_audit_syslog, isi_audit_purge_helper |
| Backend network | isi_lbfo_d |
| CloudPools | isi_cpool_d |
| Cluster monitoring | isi_array_d, isi_paxos |
| Configuration store | isi_tardis_d, isi_tardis_gcfg_d |
| DNS | isi_dnsiq_d isi_cbind_d |
| Drive | isi_drive_d, isi_drive_repurpose_d |
| Diagnostics | isi_diags_d |
| Fast delete | isi_trash_d |
| Healthchecks | isi_protohealth_d |
| IPMI management | isi_ipmi_mgmt_d |
| Migration | isi_vol_copy, isi_retore |
| NDMP Backup | isi_ndmp_d |
| NFS | isi_nfs_convert, isi_netgroup_d |
| Remote assist | isi_esrs_d, isi_esrs_api |
| SED Key Manager | isi_km_d |
| Services manager | isi_mcp_d |
| SmartLock Compliance | isi_comp_d |
| SmartSync | isi_dm_d |
| SyncIQ | siq_bandwidth, siq_generator, siq_pworker, siq_pworker_hasher, siq_stf_diff, siq_sworker, siq_sworker_hasher, siq_sworker_tmonitor, siq_coord, siq_sched, siq_sched_rotate_reports |
| Upgrade Signing | isi_catalog_init |
The ilog logging level provides for three types of capabilities:
- Severity (which maps to syslog severity)
- Special
- Custom
Plus the ilog severity level settings are as follows:
| Ilog Severity Level | Syslog Mapping |
| IL_FATAL | Maps to LOG_CRIT. Calls exit after message is logged. |
| IL_ERR | Maps to LOG_ERR |
| IL_NOTICE | Maps to LOG_INFO |
| IL_INFO | Maps to LOG_INFO |
| IL_DEBUG | Maps to LOG_DEBUG |
| IL_TRACE | Maps to LOG_DEBUG |
For example, the following CLI command will set the NDMP service to log at the ‘info’ level:
# isi_ilog -a isi_ndmp_d --level info
Note that logging levels do not work quite like syslog, as each level is separate. Specifically, if an application’s criteria set to log messages with the ‘IL_DEBUG level’ it will only log those debug messages, and not log messages at any higher severity. To log at a level and all higher severity levels, ilog allows ‘PLUS’ (–level <level>+) combination settings.
Logging configuration is per named application, not per process, and settings are managed on a per-node basis. Any cluster-wide ilog criteria changes will require the use of the ‘isi_for_array’ CLI utility.
Be aware that syslog is still the standard target for logging and /etc/mcp/templates/syslog.conf (rather than /etc/syslog.conf) is used to enable sysloging. If ‘use_syslog’ is set to true, but syslog.conf is not modified, syslog entries will not be created. When syslog is enabled, if ‘log_file’ points to the same syslog file, duplicate log entries will occur, one from syslog and one from the log file.
Other isi_log CLI commands include:
List all apps:
# isi_ilog -L
Print settings for an app:
# isi_ilog -a <service_name> -p
Set application level to info:
# isi_ilog -a <service_name> --level info
Turn off syslog logging for application:
# isi_ilog -a <service_name> --syslog off
Turn on logging to a file for a service:
# isi_ilog -a <service_name> --file /ifs/logs/<service_name>.log
Of the various services that use ilog, OneFS auditing is among the most popular. As such, it has its own configuration through the ‘isi audit’ CLI command set, or from the WebUI via Cluster management > Auditing:
Additionally, the ‘audit setting global’ CLI command allows is used to enable and disable cluster auditing, as well as configure retention periods, remote CEE and syslog services, etc.
# isi audit settings global view Protocol Auditing Enabled: Yes Audited Zones: System, az1 CEE Server URIs: - Hostname: Config Auditing Enabled: Yes Config Syslog Enabled: Yes Config Syslog Servers: 10.20.40.240 Protocol Syslog Servers: 10.20.40.240 Auto Purging Enabled: No Retention Period: 180
Additionally, the various audit event attributes can be viewed and modified via the ‘isi audit settings’ CLI command.
# isi audit settings view Audit Failure: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory Audit Success: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory Syslog Audit Events: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory Syslog Forwarding Enabled: Yes
To configure syslog forwarding, review the zone specific audit settings and ensure syslog audit events (for local) are set and syslog forwarding is enabled (for remote).
Note that the ‘isi audit settings’ CLI command defaults to the ‘system’ zone unless the ‘–zone’ flag is specified. For example, to view the configuration for the ‘az1’ access zone, which in this case is set to non-forwarding:
# isi audit settings view --zone=az1 Audit Failure: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory Audit Success: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory Syslog Audit Events: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory Syslog Forwarding Enabled: No
The cluster’s /etc/syslog.conf file should include the IP address of the server that’s being forwarded to (in this example, a Linux box at 10.20.40.240):
!audit_config *.* /var/log/audit_config.log *.* @10.20.40.240 !audit_protocol *.* /var/log/audit_protocol.log *.* @10.20.40.240
Output on the remote host will be along the lines of:
Jul 31 17:46:40 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|OPEN|SUCCESS|1442207|FILE|CREATED|4314890714|/ifs/test/audit_test2.doc Jul 31 17:46:43 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|CLOSE|SUCCESS|FILE|0:0|0:0|4314890714|/ifs/test/audit_test2.doc Jul 31 17:46:43 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|OPEN|SUCCESS|129|FILE|OPENED|4314890714|/ifs/test/audit_test2.doc Jul 31 17:46:43 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|CLOSE|SUCCESS|FILE|0:0|0:0|4314890714|/ifs/test/audit_test2.doc.txt Jul 31 17:46:43 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|RENAME|SUCCESS|FILE|4314890714|/ifs/test/ audit_test2.doc.txt|/ifs/test/audit_test.txt Jul 31 17:46:44 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|OPEN|FAILED:3221225524|129|FILE|DOES_NOT_EXIST||/ifs/test/audit_test2.doc Jul 31 17:46:45 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|CLOSE|SUCCESS|FILE|0:0|0:0|4314890714|/ifs/test/audit_test2.doc Jul 31 17:46:45 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|OPEN|SUCCESS|1179785|FILE|OPENED|4314890714|/ifs/test/audit_test3.txt Jul 31 17:46:45 isln-tme-1 (id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|CLOSE|SUCCESS|FILE|0:0|0:0|4314890714|/ifs/test/audit_test3.txt Jul 31 17:46:45 isln-tme-1 syslogd last message repeated 6 times Jul 31 17:46:51 isln-tme-1 (id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|OPEN|SUCCESS|1180063|FILE|OPENED|4314890714|/ifs/test/audit_test3.txt Jul 31 17:46:51 isln-tme-1 (id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|CLOSE|SUCCESS|FILE|0:0|0:0|4314890714|/ifs/test/audit_test3.txt Jul 31 17:46:51 isln-tme-1(id1) audit_protocol[2188] S-1-22-1-0|0|System|1|10.20.40.1|SMB|CLOSE|SUCCESS|FILE|0:0|5:1|4314890714|/ifs/test/audit_test3.txt