SupportAssist, Dell’s remote connectivity system, gets an enhancement in OneFS 9.9 with the addition of support for IPv6 network environments.
Within OneFS, SupportAssist is intended for transmitting events, logs, and telemetry from PowerScale to Dell support. Helping to rapidly identify, diagnose, and resolve cluster issues, SupportAssist drives productivity improvements by replacing manual routines with automated support. Improved time to resolution, or entire avoidance, is boosted by predictive issue detection and proactive remediation. Additionally, SupportAssist is included with all PowerScale support plans – although the features may vary based on service level agreement (SLA).
Delivering a consistent remote support experience across the Dell storage portfolio, SupportAssist can be of considerable benefit to any site that can send telemetry off-cluster to Dell over the internet.
SupportAssist’s remote information connectivity engine, or RICE, integrates the Dell Embedded Service Enabler (ESE) into OneFS along with a suite of daemons to allow its use on a distributed system. SupportAssist uses the Dell Connectivity Hub and can either interact directly, or through a Secure Connect gateway.
At its core, SupportAssist comprises a variety of components that gather and transmit various pieces of OneFS data and telemetry to Dell Support, via the Embedded Service Enabler (ESE). These workflows include CELOG events, In-product activation (IPA) information, CloudIQ telemetry data, isi-gather-info (IGI) logsets, and provisioning, configuration and authentication data to ESE and the various backend services.
| Workflow | Details |
| CELOG | SupportAssist can be configured to send CELOG events and attachments via ESE to CLM. CELOG has a ‘supportassist’ channel that, when active, will create an EVENT task for SupportAssist to propagate. |
| License Activation | The isi license activation start command uses SupportAssist to connect.
Several pieces of PowerScale and OneFS functionality require licenses, and to register and must communicate with the Dell backend services in order to activate those cluster licenses. SupportAssist is the preferred mechanism to send those license activations via the Embedded Service Enabler(ESE) to the Dell backend. License information can be generated via the ‘isi license generate’ CLI command, and then activated via the ‘isi license activation start’ syntax. |
| Provisioning | SupportAssist must register with backend services in a process known as provisioning. This process must be executed before the Embedded Service Enabler(ESE) will respond on any of its other available API endpoints. Provisioning can only successfully occur once per installation, and subsequent provisioning tasks will fail. SupportAssist must be configured via the CLI or WebUI before provisioning. The provisioning process uses authentication information that was stored in the key manager upon the first boot. |
| Diagnostics | The OneFS isi diagnostics gather and isi_gather_info logfile collation and transmission commands have a –supportassist option. |
| Healthchecks | HealthCheck definitions are updated using SupportAssist. |
| Telemetry | CloudIQ Telemetry data is sent using SupportAssist. |
| Remote Support | Remote Support uses SupportAssist and the Connectivity Hub to assist customers with their clusters. |
SupportAssist requires an access key and PIN, or hardware key, in order to be enabled, and secure keys are held in a secure key manager under the RICE domain.
In addition to the transmission of data from the cluster to Dell, Connectivity Hub also allows inbound remote support sessions to be established for remote cluster troubleshooting.
In OneFS 9.9, SupportAssist adds the ability to set IPv6 addresses for a gateway host and backup gateway host via the gateway connectivity option. It also allows the selection of one or more IPv6-family subnets and pools.
From the OneFS command line interface, SupportAssist is configured through the ‘isi supportassist’ command set, to which OneFS 9.9 adds additional parameters and options in support of IPv6 networking.
To configure SupportAssist for IPv6 from the CLI, select one or more static IPv6 subnets/pools for outbound communication:
# isi supportassist settings modify --network-pools="ipv6subnet.ipv6pool"
A direct connection to Support can be configured with the following syntax:
# isi supportassist settings modify --connection-mode direct
Alternatively, connectivity can be configured via a Secure Connect Gateway as follows:
# isi supportassist settings modify --connection-mode gateway # isi supportassist settings modify --gateway-host <IPv6 or FQDN>
Similarly for a backup gateway:
# isi supportassist settings modify –-backup-gateway-host <IPv6 or FQDN>
The following CLi syntax can be used to provision SupportAssist using a hardware key (if present):
# isi supportassist provision start
Note that new cluster nodes shipped after January 2023 should already have a built-in hardware key
For older clusters containing nodes without hardware keys, SupportAssist can be provisioned using an access key and pin as follows:
# isi supportassist provision start –access-key <key> --pin <pin>
The access key and pin can be obtained from the self-service E-support portal on the Dell Support site.
Alternatively, SupportAssist provisioning can also be performed via the WebUI wizard by navigating to Cluster Management > General Settings > SupportAssist > Connect SupportAssist:
Finally, there are a few SupportAssist IPv6 caveats and considerations that are worth noting. These include:
| Area | Caveat / Consideration |
| Networking | • Cannot mix IPv4-family and IPv6-family subnets and pools
• Either use all-IPv4 or all-IPv6 networking, with the possibility to switch network families later • Choosing subnets and pools enforces FQDN hostnames or IPv4/IPv6 address validation for gateway host and backup gateway host |
| Security | • SupportAssist with direct connectivity requires network ports 443 and 8443 to be open between the cluster and Dell Support. |
| Compatibility | • ESRS is permanently disabled once SupportAssist is enabled on a cluster. |
| Enablement | • SupportAssist for IPv6 networks becomes ready to provision after OneFS 9.9 upgrade commit.
• OneFS 9.8 and earlier releases only allow IPv4 networking for SupportAssist. • On-cluster health checks inform on the best way to migrate to SupportAssist |
Under the hood, the OneFS SupportAssist relies on the following infrastructure and services:
| Service | Name |
| ESE | Embedded Service Enabler. |
| isi_rice_d | Remote Information Connectivity Engine (RICE). |
| isi_crispies_d | Coordinator for RICE Incidental Service Peripherals including ESE Start. |
| Gconfig | OneFS centralized configuration infrastructure. |
| MCP | Master Control Program – starts, monitors, and restarts OneFS services. |
| Tardis | Configuration service and database. |
| Transaction journal | Task manager for RICE. |
ESE, isi_crispies_d, isi_rice_d, and the Transaction Journal are exclusive to SupportAssist, whereas gconfig, MCP, and tardis are used by multiple other OneFS components.
The remote information connectivity engine (RICE) represents the SupportAssist ecosystem for OneFS to connect to the Dell backend, and the basic architecture is as follows:
The Embedded Service Enabler (ESE) is at the core of the connectivity platform and acts as a unified communications broker between the PowerScale cluster and Dell Support. ESE runs as a OneFS service and, on startup, looks for an on-premises gateway server, such as SupportAssist Enterprise. If none is found, it connects back to the connectivity pipe (SRS). The collector service then interacts with ESE to send telemetry, obtain upgrade packages, transmit alerts and events, etc.
Depending on the available resources, ESE provides a base functionality with optional capabilities to enhance serviceability as appropriate. ESE is multithreaded, and each payload type is handled by different threads. For example, events are handled by event threads, and binary and structured payloads are handled by web threads. Within OneFS, ESE gets installed to /usr/local/ese and runs as the ese user in the ese group.
The responsibilities of isi_rice_d include listening for network changes, getting eligible nodes elected for communication, monitoring notifications from CRISPIES, and engaging Task Manager when ESE is ready to go.
The Task Manager is a core component of the RICE engine. Its responsibility is to watch the incoming tasks that are placed into the journal and assign workers to step through the tasks state machine until completion. It controls the resource utilization (python threads) and distributes tasks that are waiting on a priority basis.
The ‘isi_crispies_d’ service exists to ensure that ESE is only running on the RICE active node, and nowhere else. It acts, in effect, like a specialized MCP just for ESE and RICE-associated services, such as IPA. This entails starting ESE on the RICE active node, re-starting it if it crashes on the RICE active node, and stopping it and restarting it on the appropriate node if the RICE active instance moves to another node. We are using ‘isi_crispies_d’ for this, and not MCP, because MCP does not support a service running on only one node at a time.
The core responsibilities of ‘isi_crispies_d’ include:
- Starting and stopping ESE on the RICE active node
- Monitoring ESE and restarting, if necessary. ‘isi_crispies_d’ restarts ESE on the node if it crashes. It will retry a couple of times and then notify RICE if it’s unable to start ESE.
- Listening for gconfig changes and updating ESE. Stopping ESE if unable to make a change and notifying RICE.
- Monitoring other related services.
The state of ESE, and of other RICE service peripherals, is stored in the OneFS tardis configuration database so that it can be checked by RICE. Similarly, ‘isi_crispies_d’ monitors the OneFS Tardis configuration database to see which node is designated as the RICE ‘active’ node.
The ‘isi_telemetry_d’ daemon is started by MCP and runs when SupportAssist is enabled. It does not have to be running on the same node as the active RICE and ESE instance. Only one instance of ‘isi_telemetry_d’ will be active at any time, and the other nodes will be waiting for the lock.
So there you have it – PowerScale SupportAssist and it’s newly minted IPv6 networking enhancement in OneFS 9.9.