Month: June 2022

In the first blog of this series, we looked at OneFS SmartSync’s architecture and attributes. Next, we’ll delve into the configuration side of things, and walk through a basic setup.

Since there’s no SmartSync WebUI yet in OneFS 9.4, the bulk of the SmartSync configuration is performed via the ‘isi dm’ CLI tool, which contains the following the principal subcommands:

Subcommand	Description
isi dm accounts	Manage Datamover accounts. An activate SyncIQ license is required to create Datamover accounts.
isi dm base-policies	Manage Datamover base-policy. Base policies are templates to provide common values to groups of related concrete Datamover policies. Eg. Define a base policy to override the run schedule of a concrete policy.
isi dm certificates	Manage Datamover certificates.
isi dm config	Show Datamover Manual Configuration.
isi dm datasets	Show Datamover Dataset Information.
isi dm historical-jobs	Manage Datamover historical jobs.
isi dm jobs	Manage Datamover jobs.
isi dm policies	Manage Datamover policy. Policies can be either: CREATION – Creates/replicates a dataset, either once or on a schedule. COPY – Defines a one-time copy of a dataset to or from a remote system
isi dm throttling	Manage Datamover bandwidth and CPU throttling. Bandwidth throttling rules can be configured for each Datamover job.

The high-level view of the SmartSync setup and configuration process is as follows:

 

1. The first step involves installing or upgrading the cluster to OneFS 9.4. SmartSync replication is handled by the ‘isi_dm_d’ service, which is disabled by default and needs to be enabled prior to configuring and using SmartSync. This can be easily accomplished with the following CLI syntax:

# isi services -a isi_dm_d

Service 'isi_dm_d' is disabled.

# isi services -a isi_dm_d enable

The service 'isi_dm_d' has been enabled.

 

2. SmartSync uses TLS (transport layer security, or SSL) and, as such, requires trust to be established between the source and target clusters. In addition to a Certificate Authority (CA) and Certificate Identity (CI) for authorization and authentication, both clusters also require encryption to be enabled in order for the isi_dm_d service to run. The best practice is to use a local CA to sign each cluster’s CI, but self-signed certificates can be used instead in the absence of a suitable CA.

Before creating accounts, certificates must be generated and copied to the appropriate clusters. The following Certificate Authorities (CA) and trust hierarchies are required:

Requirement	Description
TLS certificates	● A mutually authenticated TLS handshake is required. Authorization, authentication, and encryption are provided by TLS certificates. ● TLS certificates are always required for daemon startup and all communication between Datamover engines. ● Encryption can be disabled, but authorization and authentication cannot be disabled.
Certificate Authorities (CA)	● One or more Certificate Authorities (CA) are required on each Datamover system. ● Dell recommends that customers use a new, Datamover-specific CA for signing Datamover identity certificates. ● The CA that signs an identity certificate does not need to be installed on the system that the identity certificate is installed on. Two systems trust each other if they have the CAs that signed each other’s identity certificates.
Identity certificates	● The certificate that provides authentication of the identity claimed. ● Exactly one identity certificate must exist on each Datamover system. ● Identity certificates are signed by one of the CAs deployed on the systems that the system is going to communicate with.
Trust hierarchies	● Two systems trust each other if they have the CAs that signed each other’s identity certificates. ● There is no concept of unidirectional trust—trust is entirely mutual.

The following steps can be used to generate and copy the pertinent TLS certificates to the source and target Datamover clusters:

Step	Cluster	Action	Commands
1	Source	Generate Certificate Authority (CA).	# openssl genrsa -out ca-s.key 4096 # openssl req -x509 -new -nodes -key ca-s.key -sha256 -days 1825 -out ca-s.pem
2	Source	Copy source cluster’s CA to target cluster.	# scp ca-s.pem [Target Cluster IP]:/:/root
3	Source	Generate Certificate Identity (CI).	# openssl genrsa -out identity-s.key 4096 # openssl req -new -key identity-s.key -out identity-s.csr
4	Source	Create a CI on source cluster.	# cat << EOF > identity-s.ext authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage=digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment EOF
5	Source	Sign source cluster’s CI with source cluster’s CA.	# openssl x509 -req -in identity-s.csr -CA ca-s.pem -CAkey ca-s.key -CAcreateserial -out identity-s.crt -days 825 -sha256 -extfile identity-s.ext
6	Target	Generate a CA on target cluster.	# openssl genrsa -out ca-t.key 4096 # openssl req -x509 -new -nodes -key ca-t.key -sha256 -days 1825 -out ca-t.pem
7	Target	Copy target cluster CA to source cluster.	# scp ca-t.pem [Source Cluster IP]:/root
8	Target	Generate CI on target cluster.	# openssl genrsa -out identity-t.key 4096 # openssl req -new -key identity-t.key -out identity-t.csr
9	Target	Create a CI on target cluster.	# cat << EOF > identity-t.ext authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage=digitalSignature,nonRepudiation,keyEncipherment,dataEncip herment EOF
10	Target	Sign this CI with target cluster’s CA.	# openssl x509 -req -in identity-t.csr -CA ca-t.pem -CAkey ca-t.key -CAcreateserial -out identity-t.crt -days 825 – sha256 -extfile identity-t.ext

 

3. Next, the various CAs and CIs are installed across the two clusters.

Step	Cluster	Action	Command
1	Source	Install source cluster’s CA.	# isi dm certificates ca create “$PWD”/ca-s.pem –name <source-cluster-ca>
2	Source	Install target cluster’s CA.	# isi dm certificates ca create “$PWD”/ca-t.pem –name <target-cluster-ca>
3	Source	Install source cluster’s CI.	# isi dm certificates identity create “$PWD”/identity-s.crt –certificate-key-path “$PWD”/identity-s.key –name <source-cluster-identity>
4	Target	Install target cluster’s CA.	# isi dm certificates ca create “$PWD”/ca-t.pem –name <target-cluster-ca>
5	Target	Install source cluster’s CA.	# isi dm certificates ca create “$PWD”/ca-s.pem –name <source-cluster-ca>
6	Target	Install target cluster’s CI.	# isi dm certificates identity create “$PWD”/identity-t.crt –certificate-key-path “$PWD”/identity-t.key –name <target-cluster-identity>

Note that the certificates must be located under /ifs when performing the import, otherwise an error similar to the following will be returned:

Invalid certificate path: /root/ca-s.pem [CERTS_CERT_INVALID]

At this point, encryption is now configured on the source and target clusters.

 

4. By default, a local account, ‘DM Local Account’, is already configured. The ‘isi dm accounts list’ command can be used to display this ‘DM Local Account.’

# isi dm accounts list

ID                                               Name             URI             Account Type  Auth Mode   Local Network Pool  Remote Network Pool

----------------------------------------------------------------------------------------------------------------------------------------------------

0060167118de5018ab62800ce595db9bdb40000000000000 DM Local Account dm://[::1]:7722 DM            CERTIFICATE

----------------------------------------------------------------------------------------------------------------------------------------------------

Total: 1

The following steps illustrate configuring a push policy from the source cluster. Note that a single account can be used for both a push and pull policy, depending on the replication topology. After encryption is configured, the next step is to add a replication account to the source cluster, pointing replication to a target cluster.

On the source cluster, add a replication account using the ‘isi dm accounts create’ CLI command:

# isi dm accounts create DM dm://[Target Cluster IP]:7722 [‘target-acct’]

If desired, local and remote SmartConnect pools can be specified for the source and target clusters, respectively, with the –local-network-pool and –remote-network-pool flags.

The ‘isi dm accounts list’ command can be used to verify successful account creation:

# isi dm accounts list

ID                                               Name             URI             Account Type  Auth Mode   Local Network Pool  Remote Network Pool

----------------------------------------------------------------------------------------------------------------------------------------------------

f8f21e66c32476412b621d182495f22d3e31000000000000 DM Local Account dm://[::1]:7722 DM            CERTIFICATE

000c38b4ga22e3810d53ff27449b285b98c8000000000000 rmt-acct          dm://10.20.50.130:7722 DM

----------------------------------------------------------------------------------------------------------------------------------------------------

Total: 2

In the above, the ‘DM Local Account’ is the source cluster’s account, and ‘rmt-acct’ is the target cluster’s account, plus IP address.

 

5. Two policies are needed here. First, the ‘isi dm policies create’ CLI command can be run with the ‘CREATION’ policy option in order to create a dataset. The syntax for this command to run at ‘normal’ priority is:

# isi dm policies create [Policy Name] NORMAL true CREATION -- creation-account-id=[DM local account] --creation-base-path= -- creation-dataset-retention-period= --creation-dataset-reserve= -- creation-dataset-expiry-action=DELETE ––recurrence=”cron expression” --start-time="YYYY-MM-DD HH:MM:SS"

The configuration parameters for the ‘isi dm policies create’ command include:

Parameter	Description
policy-type	Specifies the type of policy. Options are: ● CREATION —the process of creating the dataset ● COPY —used for one-time data transfers ● REPEAT_COPY —used for repeated transfers ● EXPIRATION —how long the snapshot is stored
priority	Assigns a priority to this policy. The options are: LOW | NORMAL | HIGH.
true	Specifies that the policy is enabled.
creation-account-id	The DM local account ID specified in the isi dm accounts list command.
creation-base-path	For SmartSync this specifies the directory path or file for the dataset. For cloud copy, this specifies the object store key prefix.
creation-dataset-retention-period	How long the dataset is retained in seconds before expiration.
creation-dataset-reserve	how many datasets to keep in reserve that are protected from expiration, irrespective of the creation-dataset-retention-period.
creation-dataset-expiry-action	Specifies what happens with the dataset after expiration. With OneFS 9.4, the only expiration option is DELETE.
recurrence	How often the policy runs.
Start-time	The date and time when the policy runs. If a prior date is entered, the policy runs immediately.

The following CLI command creates a Datamover CREATION policy named createTestDataset. The policy creates a dataset with the base filepath /ifs/test/dm/data1. The creation account is the local Datamover account. The dataset expires 1,500 seconds (25 minutes) after its creation, after which it is deleted. The policy starts running June 1, 2022, at 12pm.

# isi dm policies create --name=createTestDataset --enabled=true --priority=low --policy-type=CREATION --creation-base-path=/ifs/test/dm/data1 --creation-account-id=local --creation-dataset-expiry-action=DELETE --creation-dataset-retention-period=1500 --start-time "2022-12-01 12:00:00"

To list the Datamover policies:

# isi dm policies list

ID   Validity  Name              Enabled  Disabled By DM  Priority  Policy Type  Base Policy ID  Date Times  Recurrence  Start Time          Parent Exec Policy ID

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

1 Yes createTestDataset Yes No LOW    CREATION - - - 2022-12-01 12:00:00 -                 

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

The ‘isi dm policies view’ CLI syntax can be used to inspect details of a policy, in this case ‘createTestDataset’ with an ID of 1 above:

# isi dm policies view 1

                        ID: 1

                  Validity: Yes

                      Name: createProdDataset

                   Enabled: Yes

            Disabled By DM: No

                  Priority: LOW

                   Run Now: No

            Base Policy ID: -

     Parent Exec Policy ID: -

                  Schedule

                    Date Times: -

                    Recurrence: -

                    Start Time: 2022-12-01 12:00:00

Policy Specific Attributes

                        Policy Type: CREATION

                    Creation Policy

                           Account ID: local

                            Base Path: /ifs/test/dm/data1

                            Retention

               Dataset Retention Period: 1500

                        Dataset Reserve: 2

                  Dataset Expiry Action: DELETE

In addition to the newly configured CREATION policy, a COPY policy is also required to perform the data move. This can be created as follows:

# isi dm policies create archive-restore NORMAL true COPY --copy-source-base-path=/ifs/test/dm/data1 --copycreate-dataset-on-target=true --copy-base-base-account-id= f8f21e66c32476412b621d182495f22d3e31000000000000 --copy-base-source-accountid= f8f21e66c32476412b621d182495f22d3e31000000000000--copy-base-target-account-id=000c38b4ga22e3810d53ff27449b285b98c8000000000000--copy-base-target-basepath=/ifs/test/dm/data1 --copy-base-target-dataset-type=FILE --copy-base-dataset-retention-period=3600 --copy-base-dataset-reserve=2 --copy-base-policy-dataset-expiry-action=DELETE

Confirm both the COPY and CREATION Datamover policies are present:

# isi dm policies list

ID   Validity  Name              Enabled  Disabled By DM  Priority  Policy Type  Base Policy ID  Date Times  Recurrence  Start Time          Parent Exec Policy ID

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

1 Yes createTestDataset Yes No LOW    CREATION -      -     -     -                 

2 Yes archive-restore   Yes No NORMAL COPY     -      -     -     –

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

6. The next step is to run the CREATION policy (ID = 1) in order to create the dataset:

# isi dm policies modify 1 --run-now=true

The running job can be inspected as follows:

# isi dm jobs list

ID Job Type Job Priority Job Policy ID Job Control Request Job Start Time Job End Time Job State Job State Flags

----------------------------------------------------------------------------------------------------------------

201 DATASET_CREATION_JOB NORMAL  1  NONE  2022-06-23T14:52:22 2022-06-23T14:53:04 finishing   No failure

----------------------------------------------------------------------------------------------------------------

Total: 1

Once the job has completed, the ‘isi dm historical-jobs list’ CLI command allows the dataset creation policy’s status to be queried.

# isi dm historical-jobs list

ID Job Type Job Priority Job Policy ID Job Control Request Job Start Time Job End Time Job State Job State Flags

----------------------------------------------------------------------------------------------------------------

201 DATASET_CREATION_JOB NORMAL  1  NONE  2022-06-23T14:52:22 2022-06-23T14:54:51 finished   No failure

----------------------------------------------------------------------------------------------------------------

Total: 1

Finally, run the COPY policy (ID = 2) to replicate the dataset from the source to target cluster:

# isi dm policies modify 2 --run-now=true

# isi dm jobs list

ID Job Type Job Priority Job Policy ID Job Control Request Job Start Time Job End Time Job State Job State Flags

----------------------------------------------------------------------------------------------------------------

202 DATASET_BASELINE_COPY_JOB NORMAL  2  NONE   2022-06-23T14:55:11 2022-06-23T14:56:48 running   No failure

----------------------------------------------------------------------------------------------------------------

Total: 1

When the COPY job has completed, the ‘historical-jobs’ output will now show both the CREATION and COPY job details:

# isi dm historical-jobs list

ID Job Type Job Priority Job Policy ID Job Control Request Job Start Time Job End Time Job State Job State Flags

----------------------------------------------------------------------------------------------------------------

202 DATASET_BASELINE_COPY_JOB NORMAL  2  NONE   2022-06-23T14:55:11 2022-06-23T14:57:06 finished   No failure

201 DATASET_CREATION_JOB NORMAL  1  NONE  2022-06-23T14:52:22 2022-06-23T14:54:51 finished   No failure

----------------------------------------------------------------------------------------------------------------

Total: 2

Once created, the new dataset can be inspected via the ‘isi dm datasets list’ command output:

# isi dm datasets list

ID Dataset State Dataset Type Dataset Base Path Dataset Subpaths Dataset Creation Time Dataset Expiry Action Dataset Retention Period

-------------------------------------------------------------------------------------------------------------------------------------

1  COMPLETE FILE  /ifs/test/dm/data1   - 2022-06-23T14:54:51  DELETE  2022-06-23T15:54:51

-------------------------------------------------------------------------------------------------------------------------------------

Total: 1

To view Datamover policies:

# isi dm policies view

Note that the procedure above configures push replication of a dataset from a source to target. Conversely, to perform a pull from the target cluster, the replication account is instead added to the target cluster, and with the source cluster’s IP used:

# isi dm accounts create DM dm://[Source Cluster IP]:7722 [‘source-acct’]

Object data replication to public cloud or Dell ECS targets can also be configured with the ‘isi dm accounts create’ CLI command, but does require a couple of additional parameters, namely:

Parameter	Description
Object store type	AWS_S3, Azure, or ECS_S3
URI	{http,https}://hostname:port/bucketname
Auth	Access ID, Secret Key
Proxy	Optional proxy information

For example:

# isi dm accounts create AWS_S3 https://aws-host:5555/bucket dm-account-name --authmode CLOUD --access-id aws-access-id --secret-key aws-secret-key

Be aware that a dataset must be available before a copy, or repeat-copy data replication policy runs, or the policy will fail.

Behind the scenes, dataset creation leverages a SnapshotIQ snapshot, which can be inspected via the ‘isi snapshot list’ command. These DM dataset snapshots are easily recognizable due to their ‘isi_dm’ prefixed naming convention.

In the final article in this series, we’ll take a look at SmartSync management, monitoring, and troubleshooting.

Amongst the bevy of new functionality introduced in OneFS 9.4 is SmartSync v1, and we’ll be taking a look at this new replication product over the course of the next couple of blog articles.

So, the new SmartSync Datamover enables flexible data movement and copying, incremental resyncs, and push and pull data transfer of file data between PowerScale clusters. Additionally, SmartSync CloudCopy also enables the copying of file-to-object data from a source cluster to a cloud object storage target. Cloud object targets include AWS S3 and Microsoft Azure, as well as Dell ECS.

Having a variety of target destination options allows multiple copies of a dataset to be stored across locations and regions, both on and off-prem, providing increased data resilience and the ability to rapidly recover from catastrophic events.

CloudCopy uses HTTP as the data replication transport layer to cloud storage, while cluster to cluster SmartSync leverages a proprietary RCP-based messaging system. In addition to the replication of the actual data, SmartSync also preserves the common file attributes including Windows ACLs, POSIX permissions and attributes, creation times, extended attributes, alternate data streams, etc.

In order to use SmartSync, SyncIQ  must be licensed and active across all nodes in the cluster. Additionally, a cluster account with the ISI_PRIV_DATAMOVER privilege is needed in order to configure and run SmartSync data mover policies. While file-to-file replication requires SmartSync to be running on both source and target clusters, for OneFS Cloud Copy to transfer to/from cloud storage, only the OneFS 9.4 cluster requires the SmartSync data mover. No data mover needed on the cloud systems. Be aware that the inbound TCP 7722 IP port must be open across any intermediate gateways and firewalls to allow SmartSync replication to occur.

Under the hood, replication is handled by the ‘isi_dm_d’ service, which is disabled by default, and needs to be enabled prior to configuring and using SmartSync. SmartSync uses TLS (transport layer security, or SSL) and, as such, requires trust to be established between the source and target clusters. In addition to a Certificate Authority (CA) and Certificate Identity (CI) for authorization and authentication, both clusters also require encryption to be enabled in order for the isi_dm_d service to run. The best practice is to use a local CA to sign each cluster’s CI,  but self-signed certificates can be used instead in the absence of a suitable CA.

The SmartSync Datamover has a purpose-build, integrated job execution engine, and Datamovers are executed on each cluster node in cooperative mode.

Shared Key-Value Stores (KVS) are used for jobs/tasks distribution, and extra indexing is implemented for quick lookups by task state, task type, and alive time. There are no dependencies or communication between tasks, and job cancellation and pausing is handled by posting a ‘request’ into a job record (request polling).

Within the SmartSync hierarchy, accounts define the connections to remote systems, policies define the replication configurations, and jobs perform the work:

Component	Details
Accounts	Datamover accounts: –          URI, eg. dm://remotenas.isln.com:7722 –          Local and remote network pools defining nodes/interfaces to use for data transfer –          Client and server certificates to enable TLS CloudCopy accounts: –          Account type (AWS S3, ECS S3, Azure) –          URI, eg. https://cloudcluster.isln.com:9002/cloudbucket –          Credentials
Policies	–          Dataset creation policy –          Dataset copy policy –          Dataset repeat copy policy –          Dataset expiration policy
Jobs	Runtime entities created based on policies schedules. There are two major types of data transfer jobs: –         Baseline jobs for initial transfers and –         Incremental jobs for subsequent transfers between FILE Datamover systems.
Tasks	Spawned by jobs and are the individual chunks of work that a job must perform. No 1-to-1 relationship to their associated files.

 

SmartSync Datasets are self-contained, independent entities. Once created, they’re assigned globally-unique IDs, and backed by file system snapshots on PowerScale. Parent-child relationships are used for incremental transfers, and a handshake determines the exact changeset to be transferred.

As demand for replication on a source cluster increases, the additional compute and network load needs to be considered. Multiple targets can generate a significant demand on the source cluster, with replication traffic contending with client workloads as data is pushed to the target. Fortunately, SmartSync allows a target cluster to pull the dataset, thereby minimizing the resource impacts on the source.

For single-source dataset environments with numerous targets, push replication can be incredibly useful, allowing the source cluster’s resources to be focused on client IO. In addition to both push and pull replication, SmartSync also supports a variety of topologies, such as fan-out, chaining, etc.

SmartSync provides enhanced replication failure resilience, minimizing replication times even when a job runs into an error. Rather than failing an entire replication job if an error is encountered, requiring a manual restart, SmartSync instead places the job into a paused state, and presents three options:

1. Cancel the job altogether.
2. Resolve the errors and resume the job.
3. Complete a partial replication.

With option 3, the portion of the dataset already transferred is retained, thereby decreasing the subsequent job’s work and execution time.

The SmartSync architecture intentionally decouples source cluster snapshot creation (dataset creation) from the actual data replication transfer to the target, allowing each to run independently via separate configured policies configured for each. This helps mitigate the disruptive chain effect of a failure during the snapshot process early in the process. Additionally, SmartSync offers parent-child policies which launch a replication job only after successful snapshot creation, providing an alternative to recurrence in situations where it’s unclear how long a previous policy may take to complete.

With SmartSync, ‘re-baselining’ (full-resync) is not required for source-target clusters which already contain an earlier version  of a dataset. For example, in the following three-cluster DR topology, cluster A replicates to B, and B replicates to C:

A parent-child relationship means that, if cluster B becomes unavailable, the cluster A to C policy would not require a new baseline. Instead, clusters A and C’s datasets are compared via a handshake, enabling only the changed data blocks to be transferred, thereby minimizing replication overhead. This is particularly beneficial for environments with large datasets, significantly shrinking RPO and RTO times and increasing DR readiness.

When setting up a SmartSync 3-way relationship, be sure to use a single dataset creation policy when configuring datasets on the same path. If there are separate dataset creation policies for each relationship, B and C will have different datasets (snapshots) with different ​dataset IDs.  In this case, if A dies it would be impossible to establish an incremental sync relationship between B on C on those datasets, since the incremental transfer won’t be able to ‘connect’ the dataset IDs between B and C.

SmartSync allows subsequent incremental data movement by managing and re-transferring failed file transfers. Similarly, Dataset reconnect enables systems with common base datasets to establish instant incremental syncs. SmartSync also proactively locks the SnapshotIQ snapshots it generates, providing better protection and separation between Datamover and other cluster snapshots.

Other SmartSync features and functionality includes:

Feature	Details
Bandwidth throttling	Set of netmask rules. Limits are per-node.
CPU throttling	Allowed and Backoff CPU percentages.
Base policies	Template providing common values to groups of related policies (schedule, source base path, enable/disable, etc). Ie. Disabling base policy affects all linked concrete policies.
Concrete policy	Predefined set of fields from the base policy
Incremental reconnect	Ability to run incrementals between systems with common base datasets but no prior replication relationship
Unconnected nodes (NANON)	Active accounts are monitored by each node. No work allocation to nodes without network access.
Snapshot locking	Avoids accidental snapshot deletion, with subsequent re-base-lining.

 

SmartSync allows subsequent incremental data movement by managing and re-transferring failed file transfers. Similarly, Dataset reconnect enables systems with common base datasets to establish instant incremental syncs. SmartSync also proactively locks the SnapshotIQ snapshots it uses, providing better separation between Datamover and other snapshots.

Performance-wise, SmartSync is powered by a scalable run-time engine, spanning the cluster, and which spins up threads (fibers) on demand and uses asynchronous IO to process replication tasks (chunks). Batch operations are used for efficient small file, attribute, and data block transfer. Namespace contention avoidance, efficient snapshot utilization, and separation of dataset creation from transfer are salient design features of the both the baseline and incremental sync algorithms. Plus, the availability of a pull transfer model can significantly reduce the impact on a source cluster, if needed.

The streamlined baseline and incremental file transfer jobs operates as follows:

On the CloudCopy side, the SmartSync copy format provides both regular file representation, browsability and usability of file system data in the cloud. That said, as compared to the file-to-file Datamover, there are certain CloudCopy considerations and limitations to be aware of, such as no incremental copy. These also include:

CloudCopy Caveats	Details
ADS files	Skipped when encountered.
Hardlinks	An object will be created for each link (ie. links are not preserved).
Symlinks	Skipped when encountered.
Directories	An object is created for each directory.
Special files	Skipped when encountered.
Metadata	Only POSIX mode bits, UID, GID, atime, mtime, ctime are preserved.
Filename encodings	Converted to UTF-8.
Path	Path relative to root copy directory is used as object key.
Large files	An error is returned for files larger than the cloud providers maximum object size.
Long filenames	File names exceeding 256 bytes are compressed.
Long paths	Junction points are created when paths exceed 1024 bytes to redirect where objects are being stored
Sparse files	Sparse sections are not preserved and are written out fully as zeros.

As mentioned earlier, there are also some prerequisites to address before running SmartSync. First, the source and target(s) must be running OneFS 9.4 and with SyncIQ licensed across the cluster. Additionally, the identity certificates and a shared CA must be present to communicate with a peer Datamover.

In the next article in this series, we’ll turn our attention to the configuration and use of SmartSync.

In addition to the /ifs data storage partition, like most UNIX-derived operating systems, OneFS uses several system partitions, including:

Partition	Description
/	Root partition containing all the data to start up and run the system, and which contains the base OneFS software image.
/dev	Device files partition. Drives, for example, are accessed through block device files such as /dev/ad0.
/ifs	Clustered filesystem partition, which spans all of a cluster’s nodes. Includes /ifs/.ifsvar.
/usr	Partition for user programs.
/var	Partition to store variable data, such as log files, etc. In OneFS, this partition is mostly used for /var/run and /var/log.
/var/crash	The crash partition is configured for binary dumps.

One advantage of having separate partitions rather than one big chunk of space is that different parts of the OS are somewhat protected from each other. For example, if /var fills up, it doesn’t affect the root / partition.

While OneFS automatically performs the vast majority of its system housekeeping, occasionally the OneFS /var partition on one or more of a cluster’s nodes will fill up, typically as the result of heavy log writing activity and/or the presence corefile(s). If /var reaches 75%, 85%, or 95% of capacity, a CELOG event is  automatically fired and an alert sent.

The following CLI command will provide a view of /var usage across the cluster:

# isi_for_array -s "du -h /var | sort -n | tail -n10"

The typical resolution for this scenario is to rotate the logfiles under /var/log. If, after log rotation, the /var partition returns to a normal usage level, reviewing the list of recently written logs will usually determine if a specific log is rotating frequently/excessively. Log rotation will usually resolve the full-partition issue by compressing or removing large logs and old logs, thereby automatically reducing partition usage.
The ‘df -i’ CLI command, run on the node that reported the error, will display the details of the /var partition. For example:

# df -i | grep var | grep -v crash
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on
/dev/mirror/var0 1013068 49160 882864 5% 1650 139276  92% /var

If the percentage used value is 90% or higher, as above, the recommendation is to reduce the number of files in the /var partition. To remove files that do not belong in the /var partition, first run the following ‘find’ command on the node that generated the alert. This will display any files in the /var partition that are greater than 5 MB in size:

# find -x /var -type f -size +10000 -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

The output will show any large files that files that do not typically belong in the /var partition. These could include artifacts such as OneFS install packages, cluster log gathers, packet captures, or other user-created files. Remove the files or move them to the /ifs directory. If you are unsure which, if any, files are viable candidates for removal, contact Dell Support for assistance.

The ‘fstat’ CLI command is a useful tool for listing the open files on a node or in a directory, or to display files that were opened by a particular process. This information can be invaluable for determining if a process is holding a large file open. For example a node’s open files on a node can be displayed as follows:

# fstat

A list of the open files can help in monitoring the processes that are writing large files.

Using the ‘-f’flag will narrow the fstat output to a particularly directory:

# fstat -f <directory_path>

Similarly, to list the files opened by a particular process:

# fstat -p <pid>

If there are no open files found in the /var directory, it is entirely possible that a large file has become unlinked and is consuming space because one or more processes have the file open. The fstat command can be used to confirm this, as follows:

# fstat -f /var | grep var

If a process is holding a file open, output similar to the following is displayed:

root lwio 98281 4 /var 69612 -rw------- 100120000 rw

Here, the lwio daemon (PID 98281) has a 100MB file open that is approximately 100 MB (100120000 bytes). The file’s inode number, 69612, can be used to retrieve the its name:

# find -x /var -inum 69612 -print

/var/log/lwiod.log

If a process is holding a large file open and it’s inode cannot be found, the file is considered to be ‘unlinked’. In this case, the recourse is typically to restart the offending process. Note that, before stopping and restarting a process, consider any possible negative consequences. For example, stopping the OneFS SMB daemon, lwiod, in the example above would potentially disconnect SMB users.

If neither of the suggestions above resolves the issue, the logfile’s rollover file size limit can be reduced and the file itself compressed. To do this, first create a backup of the /etc/newsyslog.conf file as follows:

# cp /etc/newsyslog.conf /ifs/newsyslog.conf
# cp /etc/newsyslog.conf /etc/newsyslog.bak

Next, open the /ifs/newsyslog.conf file in emacs, vi, or editor or choice and locate the following line:

/var/log/wtmp 644 3 * @01T05 B

Change the line to:

/var/log/wtmp 644 3 10000 @01T05 ZB

These changes instruct the system to roll over the /var/log/wtmp file when it reaches 10 MB and then to compress the file with gzip. Save and close the /ifs/newsyslog.conf file, and then run the following command to copy the updated ‘newsyslog.conf’ file to the remaining nodes on the cluster:

# isi_for_array 'cp /ifs/newsyslog.conf /etc/newsyslog.conf'

If other logs are rotating frequently, or if the preceding solutions do not resolve the issue, run the isi_gather_info command to gather logs, and then contact Dell Support for assistance.

There are several options available to stop processes and create a corefile under OneFS:

CLI Command	Description
gcore	Generate a core dump file of the running process without actually killing it.
kill -6	Stop a single process and get a core dump file
killall -6	Stop all processes and get a core dump file
kill -9	Force a process to stop

The ‘gcore’ CLI command can generate a core dump file from a running process without actually killing it. First, the ‘ps’ CLI command can be used to find and display the process ID (PID) for a running process:

# ps -auxww | egrep 'USER|lsass' | grep -v grep

USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
root   68547  0.0  0.3 150464 38868 ??   S    Sun11PM   0:06.87 lw-container lsass (lsass)

In the above example, the PID for the lsass process is 68547. Next, the ‘gcore’ CLI command can be used to generate a core dump of this PID and write the output to a location of choice, in this example a file aptly named ‘lsass.core’.

 # gcore -c /ifs/data/Isilon_Support/lsass.core 68547

# ls -lsia /ifs/data/Isilon_Support/lsass.core
4297467006 58272 -rw-------     1 root  wheel  239280128 Jun 10 19:10 /ifs/data/Isilon_Support/lsass.core

Typically, the /ifs/data/Isilon_Support directory provides an excellent location to write the coredump to. Clearly, /var is not a great choice, since the partition is likely already full.

Finally, when the coredump has been written, the ‘isi_gather_info’ tool can be used to coalesce both the core file and pertinent cluster logs and the core into a convenient tarfile.

# isi_gather_info --local-only -f /ifs/data/Isilon_Support/lsass.core

# ls -lsia /ifs/data/Isilon_Support | grep -i gather
4298180122    26 -rw-r--r-- +    1 root  wheel         19 Jun 10 15:44 last_full_gather

The resulting log set, ‘/ifs/data/Isilon_Support/last_full_gather’, is then ready for upload to Dell Support for further investigation and analysis.

OneFS’ non-disruptive upgrade (NDU) functionality allows administrators to upgrade a cluster while their end user community continue to access data without error or interruption. During the OneFS rolling upgrade process, one node at a time is updated to the new code, and the active clients attached to it are automatically migrated to other nodes in the cluster. Partial upgrade is also supported, allowing a subset of cluster nodes can be upgraded, and the subset of nodes may also be grown during the upgrade. OneFS also permits an upgrade to be paused and resumed, enabling customers to span cluster upgrades over multiple smaller Maintenance Windows.

OneFS CloudPools v1.0 originally debuted back in the OneFS 8.0 release. The next major update, CloudPools v2.0, was delivered in OneFS 8.2 release, and introduced significant enhancements which included:

• Support for AWS signature authentication version 4.
• Network statistics per CloudPools account or file pool policy.
• Support for Alibaba Cloud and Amazon C2S public cloud providers.
• Full integration of CloudPools and data services like Snapshot, Sparse file handling, Quota, AVScan and WORM.
• NDMP and SyncIQ support
• Non-Disruptive Upgrade (NDU) support

CloudPools, like its SmartPools counterpart, uses the OneFS file pool policy engine to designate which data on a cluster should reside on which tier, or be archived to a cloud storage target. If files match the criteria specified in a file pool policy, the content of those files is moved to cloud storage when the job runs. Under the hood, CloudPools uses ‘SmartLink’ files within the /ifs namespace, each of which contains information about where to retrieve each file’s data blocks that have been cloud tiered. In CloudPools 1.0, these SmartLink v1 files, often referred to as ‘stubs’, do not behave like a normal file. By contrast, the SmartLink v2 files in CloudPools 2.0 are more like traditional files, each containing pointers to the CloudPools target where the data resides.

When a CloudPools 1.0 cluster is upgraded to OneFS 8.2 or later, a ‘changeover’ process is automatically initiated upon upgrade commit. This process is responsible for converting the v1 SmartLink files to v2, ensuring a seemless transition from CloudPools 1.0 to 2.0.

The following table outlines the upgrade paths available when transitioning from CloudPools 1.0 to 2.0:

Current OneFS Version	Upgrade to OneFS 8.2	Upgrade to OneFS 8.2.1 with 5/2020 RUPs	Upgrade to OneFS 8.2.2 with 5/2020 RUPs	Upgrade to OneFS 9.x
OneFS 8.0	Discouraged	Viable	Recommended	Highly recommended
OneFS 8.1	Discouraged	Viable	Recommended	Highly recommended

In a SyncIQ environment with unidirectional replication, the SyncIQ target cluster should be upgraded before the source cluster. Conversely, for bi-directional replication, the recommendation is to disable SyncIQ on both the source and target, and upgrade both clusters simultaneously.

The following CLI commands can be run on both the source and target  clusters to verify and capture their storage account, CloudPools, file pool policy, and SyncIQ configurations:

# isi cloud accounts list -v 

# isi cloud pools list -v 

# isi filepool policies list -v 

# isi sync policies list -v 

SyncIQ can be re-enabled on both source and target once the OneFS upgrades have been committed on both clusters. Be aware that the SmartLink conversion process can take considerable time, depending on the number of SmartLink files and the processing power of the target cluster.

Note that there is no need to stop the SyncIQ and/or SnapshotIQ services during the upgrade in a SyncIQ environment with unidirectional replication. However, since SyncIQ must resynchronize all converted stub files, it might take it some time to process all the changes.

The ‘isi cloud job view <job ID>’ CLI command can be used to check the status of a SmartLink upgrade process. For example, to view job ID 6:

# isi cloud job view 6

ID: 6 Description: Update SmartLink file formats

Effective State: running

Type: smartlink-upgrade

Operation State: running

Job State: running

Create Time: 2022-05-23T14:20:26

State Change Time: 2022-05-17T09:56:08

Completion Time: -

Job Engine Job: -

Job Engine State: -

Total Files: 21907433

Total Canceled: 0

Total Failed: 61

Total Pending: 318672

Total Staged: 0

Total Processing: 48

Total Succeeded: 21588652

Note that the CloudPools recall jobs will not run during an active SmartLink upgrade or conversion.

CloudPools 2.0 supports AWS signature version 4 (v4), in addition to version 2 (v2). Version 4 is generally preferred, since it provides an additional level of security.. However, be aware that any legacy CloudPools v2 cloud storage accounts cannot use v4 in the ‘upgraded’ state if the version prior to the OneFS 8.2.0 upgrade did not support V4. A patch is available for OneFS 8.1.2 to support v4 authentication, as a work-around for this issue.

While CloudPools 2.0 supports write-back in a snapshot, it does not support archiving and recalling files in the snapshot directory. If there is legacy file data in a snapshot on a cluster running a OneFS 8.1.2 or earlier, since that data consumes storage space, upon upgrade to OneFS 8.2, this snapshot storage space cannot be released since CloudPools 2.0 does not support archiving files in snapshots to the cloud.

OneFS non-disruptive upgrades can be easily managed from the WebUI by navigating to Cluster Management > Upgrade, and selecting the desired ‘Upgrade type’ from the drop-down menu. For example:

Rolling upgrades can be initiated from the OneFS CLI with the following syntax:

# isi upgrade cluster start <upgrade_image>

Since OneFS supports the ability to roll back to the previous version, in-order to complete an upgrade it must be committed.

# isi upgrade cluster commit

Up until the time an upgrade is committed, an upgrade can be rolled back to the prior version as follows.

# isi upgrade cluster rollback

The isi upgrade view CLI command can be used to monitor how the upgrade is progressing:

# isi upgrade view -i/--interactive

The following command will provide more detailed/verbose output:

# isi_upgrade_status

A faster, simpler version of isi_upgrade_status is also available in OneFS 8.2.2 and later:

isi_upgrade_node_state-a (aggregate the latest hook update for each node)-devid=<X,Y,E-F>  (filter and display by devid)-lnn=<X-Y,A,C> (filter and display by LNN)-ts (time sort entries)

If the end of a maintenance window is reached but the cluster is not fully upgraded, the upgrade process can be quiesced and then restarted using the following CLI commands:

# isi upgrade pause
# isi upgrade resume

For example:

# isi upgrade pause

You are about to pause the running process, are you sure?  (yes/[no]):

yes

The process will be paused once the current step completes.

The current operation can be resumed with the command:

`isi upgrade resume`

Note that pausing is not immediate: The upgrade will remain in a “Pausing” state until the currently
upgrading node is completed. Additional nodes will not be upgraded until the upgrade process is resumed.

The ‘pausing’ state can be viewed with the following commands: ‘isi upgrade view’ and ‘isi_upgrade_status’. Note that a rollback can be initiated either during ‘Pausing’ or ‘Paused’ states. Also, be aware that the ‘isi upgrade pause’ command has no effect when performing a simultaneous OneFS upgrade.

A rolling reboot can be initiated from the CLI on a subset of cluster nodes using the ‘isi upgrade rolling-reboot’ syntax and the ‘–nodes’ flag specifying the desired LNNs for upgrade:

# isi upgrade rolling-reboot --help

Description:

    Perform a Rolling Reboot of cluster.


Required Privileges:

    ISI_PRIV_SYS_UPGRADE


Usage:

    isi upgrade cluster rolling-reboot

        [--nodes <integer_range_list>]

        [--force]

        [{--help | -h}]


Options:

    --nodes <integer_range_list>

        List of comma (1,3,7) or dash (1-7) specified node LNNs to select. "all"

        can also be used to select all the cluster nodes at any given time.


  Display Options:

    --force

        Do not ask confirmation.

    --help | -h

        Display help for this command.

This ‘isi upgrade view’ syntax provides better visibility, status and progress of the rolling reboot process. For example:

# isi upgrade view


Upgrade Status:


Current Upgrade Activity: RollingReboot

   Cluster Upgrade State: committed

   Upgrade Process State: Not started

      Current OS Version: 9.2.0.0

      Upgrade OS Version: N/A

        Percent Complete: 0%


Nodes Progress:


     Total Cluster Nodes: 3

       Nodes On Older OS: 3

          Nodes Upgraded: 0

Nodes Transitioning/Down: 0


LNN  Progress  Version  Status

---------------------------------

1    100%        9.2.0.0  committed

2    rebooting   Unknown  non-responsive

3    0%          9.2.0.0  committed

For the longest time, the statistics from CloudPools accounts and policies were recorded but were only available via internal tools. In OneFS 9.4, these metrics are now easily accessible and presented via a new CLI ‘cloud’ option, within the familiar ’isi statistics’ CLI command set. This allows cluster administrators to gain insight into cloud accounts and policies for planning or troubleshooting CloudPools related activities.

There is no setup or configuration required in order to use CloudPools statistics, and all new and existing CloudPools accounts and policies statistics will automatically be collected and reported upon upgrade to OneFS 9.4.

The syntax for the new ‘isi statistics cloud’ CLI command is as follows:

Usage:

    isi statistics cloud <action>

        [--account <str>]

        [--policy <str>]

        [{--nodes | -n} <NODE>]

        [{--degraded | -d}]

        [--nohumanize]

        [{--interval | -i} <float>]

        [{--repeat | -r} <integer>]

        [{--limit | -l} <integer>]

        [--long]

        [--output ((Timestamp|time) | (Account|acct) | (Policy|pol) |

          (Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |

          (Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) |

          (P-ID|policy_id|id) | Node)]

        [--sort ((Timestamp|time) | (Account|acct) | (Policy|pol) |

          (Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |

          (Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) |

          (P-ID|policy_id|id) | Node)]

        [--format (table | json | csv | list | top)]

        [{--no-header | -a}]

        [{--no-footer | -z}]

        [{--verbose | -v}]

        [{--help | -h}]

The following CloudPools metrics and infomation are gathered and reported by the ‘isi statistics cloud list’ CLI command:

Name	Description
In	Bytes in.
Out	Bytes out.
Reads	Number of Reads.
Writes	Number of writes.
Deletions	Number of deletions.
Timestamp	Date and time.
GUID	Cluster global unique identifier.
Cloud	Cloud vendor.
A-Key	Cloud account key.
P-ID	Cloud policy identifier.
Node	Node number.

Standard CLI options are available for the command, including  JSON, Table, CSV output. The comprehensive list of these includes:

Option	Description
–account	Identify the account to view. Specify the account name or a phrase to match. Default is ‘all’ which will select all accounts.
–policy	Identify the policy to view. Specify the policy name or a phrase to match. Default is ‘none’ which will select no policies.
–nodes	Specify node(s) for which statistics should be reported.
–degraded	Continue to report if some nodes do not respond.
–nohumanuze	Output raw numbers without conversion to units.
–interval	Wait <INTERVAL> seconds before refreshing the display.
–repeat	Print the requested data <REPEAT> times (-1 for infinite).
–limit	Number of statistics to display.
–long	Display all possible columns.
–output	Output specified column(s): ·         Timestamp | (Account|acct) | (Policy|pol) |  (Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |  (Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) | (P-ID|policy_id|id) | Node)
–sort	Sort data by the specified comma-separated field(s) above. Prepend ‘asc:’ or ‘desc:’ to a field to change the sort order.
–format	Display statistics in table, JSON, CSV, list or top format.
–no-header	Do not display headers in CSV or table formats.
–no-footer	Do not display table summary footer information.
–verbose	Display more detailed information.
–help

In addition to sorting and filtering results, the CloudPools statistics can also be separated by node.

In its simplest form, the OneFS 9.4 ‘isi statistics cloud list’ syntax returns the following information, in this case for a three node cluster:

# isi statistics cloud list

Account Policy In    Out      Reads Writes Deletions Cloud Node

---------------------------------------------------------------

S3       0.0B  510.2KB      0      4         0  AWS   2    

S3       0.0B    0.0KB      0      0         0  AWS   1    

S3       0.0B    0.0KB      0      0         0  AWS   3    

---------------------------------------------------------------

More detailed output can be obtained by including the ‘–long’ command argument. Additionally, the ‘-r’ flag will repeat the output the number of times specified and at an interval specified by the ‘-i’ flag. For example:

# isi statistics cloud list –i 3 -–account s3 -–policy s3policy -–long –r 2

Architecturally, the new CloudPools statistics reporting infrastructure utilizes existing OneFS daemons and systems.

Under the hood, the isi_cpool_io_d service searches the CloudPools accounts and policies every three minutes and registers/unregisters them as appropriate, while new data is sent directly to the isi_cpool_sysctl service. Raw cloud metrics are passed to the isi_stats service, which generates the ordered stats keys, which are reported by the ‘isi statistics cloud list’ CLI utility and the platform API. For example:

# https://<node_ip>:8080/platform/statistics/summary/cloud

While no new log files are introduced in support of the CloudPools statistics framework, the ‘isi_gather_info’ logfile coalescing utility does now include cloud account and policy information and statistics in OneFS 9.4, which will be particularly useful for Dell Support when troubleshoot CloudPools issues

In order to access the new CloudPools statistics, all the cluster’s nodes must be running OneFS 9.4 or higher. Also, be aware of the current limitations, which include not reporting file-based statistics such as the number and size of files archived and recalled, or limiting statistics to a specific time period.

• No statistics available before OneFS 9.4 upgrade for existing CloudPools accounts since isi statistics wasn’t tracking metrics in prior releases.
• CloudPools statistics does not include cloud object cache stats. However, these can be displayed as follows:

# isi statistics query history --keys cluster.cloudpools.object_cache.stats

Or via the ‘isi_test_cpool_stats’ CLI command, for example:

# isi_test_cpool_stats -Q --objcache
------------------------------------------------
Object Cache Counters
------------------------------------------------
 object_cache_hits: 22
 object_cache_misses: 2
 object_cache_overwrite: 0
 object_cache_drop: 0
 header_cache_drop: 0
 data_cache_drop: 0
 data_cache_timeout: 0
 header_cache_timeout: 0
 object_cache_bypass: 0
{{ cmo_cache_hits: 0}}
{{ data_cache_hits: 22}}
{{ header_cache_hits: 0}}
{{ data_cache_range_hits: 22}}
{{ data_cache_range_misses: 0}}
------------------------------------------------