In this article in the Dell Technologies Connectivity Services series, we’ll dig a little deeper and look at the OneFS DTCS architecture and operation.
The OneFS Dell Connectivity Services relies on the following infrastructure and services:
| Service | Name |
| ESE | Embedded Service Enabler. |
| isi_rice_d | Remote Information Connectivity Engine (RICE). |
| isi_crispies_d | Coordinator for RICE Incidental Service Peripherals including ESE Start. |
| Gconfig | OneFS centralized configuration infrastructure. |
| MCP | Master Control Program – starts, monitors, and restarts OneFS services. |
| Tardis | Configuration service and database. |
| Transaction journal | Task manager for RICE. |
Of these, ESE, isi_crispies_d, isi_rice_d, and the Transaction Journal were introduced back in OneFS 9.5 and are exclusive to Dell Connectivity Services. In contrast, Gconfig, MCP, and Tardis are all legacy services that are employed by multiple other OneFS components.
The Remote Information Connectivity Engine (RICE) represents the Dell Connectivity Services ecosystem that allows OneFS to connect to the Dell support backend. At a high level, its architecture operates as follows:
Under the hood, the Embedded Service Enabler (ESE) is at the core of the connectivity platform and acts as a unified communications broker between the PowerScale cluster and Dell Support. ESE runs as a OneFS service and, upon startup, looks for an on-premises gateway server, such as Dell Connectivity Services Enterprise. If none is found, it connects back to the connectivity pipe (SRS). The collector service then interacts with ESE to send telemetry, obtain upgrade packages, transmit events and alerts, etc.
Depending on the available resources, ESE provides a base functionality with optional capabilities to enhance serviceability as appropriate. ESE is multithreaded, and each payload type is handled by different threads. For example, events are handled by event threads, and binary and structured payloads are handled by web threads. Within OneFS, ESE gets installed to /usr/local/ese and runs as the ese user in the ese group.
Networking-wise, Dell Connectivity Services provides full support for both IPv4 and IPv6. The responsibilities of isi_rice_d include listening for network changes, getting eligible nodes elected for communication, monitoring notifications from CRISPIES, and engaging Task Manager when ESE is ready to go.
The Task Manager is a core component of the RICE engine. Its responsibility is to watch the incoming tasks that are placed into the journal and assign workers to step through the tasks state machine until completion. It controls the resource utilization (python threads) and distributes tasks that are waiting on a priority basis.
The ‘isi_crispies_d’ service exists to ensure that ESE is only running on the RICE active node, and nowhere else. It acts, in effect, like a specialized MCP just for ESE and RICE-associated services, such as IPA. This entails starting ESE on the RICE active node, re-starting it if it crashes on the RICE active node, and stopping it and restarting it on the appropriate node if the RICE active instance moves to another node. We are using ‘isi_crispies_d’ for this, and not MCP, because MCP does not support a service running on only one node at a time.
The core responsibilities of ‘isi_crispies_d’ include:
- Starting and stopping ESE on the RICE active node
- Monitoring ESE and restarting, if necessary. ‘isi_crispies_d’ restarts ESE on the node if it crashes. It will retry a couple of times and then notify RICE if it’s unable to start ESE.
- Listening for gconfig changes and updating ESE. Stopping ESE if unable to make a change and notifying RICE.
- Monitoring other related services.
The state of ESE, and of other RICE service peripherals, is stored in the OneFS tardis configuration database so that it can be checked by RICE. Similarly, ‘isi_crispies_d’ monitors the OneFS Tardis configuration database to see which node is designated as the RICE ‘active’ node.
The ‘isi_telemetry_d’ daemon is started by MCP and runs when Dell Connectivity Services is enabled. It does not have to be running on the same node as the active RICE and ESE instance. Only one instance of ‘isi_telemetry_d’ will be active at any time, while the other nodes wait for the lock.
The current status and setup of Dell Connectivity Services on a PowerScale cluster can be queried via the ‘isi connectivity settings view’ CLI command. For example:
# isi connectivity settings view Service enabled: Yes Connection State: enabled OneFS Software ID: ELMISL08224764 Network Pools: subnet0:pool0 Connection mode: direct Gateway host: - Gateway port: - Backup Gateway host: - Backup Gateway port: - Enable Remote Support: Yes Automatic Case Creation: Yes Download enabled: Yes
This can also be viewed from the WebUI under Cluster management > General settings > Connectivity Services:
Note that once a cluster is provisioned with Dell Connectivity Services, the legacy ESRS can no longer be used.