Amongst the plethora of OneFS data services sits SmartLock, which provides immutable data storage for the cluster, guarding critical data against accidental, malicious or premature deletion or alteration. Based on a write once, read many (WORM) locking capability, SmartLock offers tamper-proof archiving for regulatory compliance and disaster recovery purposes, etc. Configured at the directory-level, SmartLock delivers secure, simple to manage data containers that remain locked for a configurable duration or indefinitely. Additionally, SmartLock satisfies the regulatory compliance demands of stringent corporate and federal data retention policies.
Once SmartLock is licensed and activated on a cluster, it can be configured to run in one of two modes:
| Mode | Description |
| Enterprise | Upon SmartLock license activation, cluster automatically becomes enterprise mode enabled, permitting Enterprise directory creation and committing of WORM files with specified retention period. |
| Compliance | A SmartLock licensed cluster can optionally be put into compliance mode, allowing data to be protected in compliance directories, in accordance with U.S. Securities and Exchange Commission rule 17a-4(f) regulations. |
Note that SmartLock’s configuration is global, so enabling it in either enterprise or compliance mode is a cluster-wide action.
Under the hood, SmartLock uses both a system clock, which is common to both modes, and a compliance clock which is exclusive to compliance mode. The latter updates the time in a protected system B-tree and, unlike the system clock, cannot be manually modified by either the ‘root’ or ‘compadmin’ roles.
SmartLock employs the OneFS job engine framework to run its WormQueue job, which routinely scans the SmartLock queue for LINs that need to be committed. By default, WormQueue is scheduled to run every day at 2am, with a ‘LOW’ impact policy and a relative job priority of ‘6’.
SmartLock also leverages the OneFS IFS domains ‘restricted writer’ infrastructure to enforce its immutability policies. Within OneFS, a domain defines a set of behaviors for a collection of files under a specified directory tree. More specifically, a protection domain is a marker which prevents a configured subset of files and directories from being deleted or modified.
If a directory has a protection domain applied to it, that domain will also affect all of the files and subdirectories under that top-level directory. A cluster’s WORM domains can be reported with the ‘isi worm domains list’.
As we’ll see, in some instances, OneFS creates protection domains automatically, but they can also be configured manually via the ‘isi worm domains create’ CLI syntax.
SmartLock domains are assigned to WORM directories to prevent committed files from being modified or deleted, and OneFS automatically sets up a SmartLock domain when a WORM directory is created. That said, domain governance does come with some boundary caveats. For example, SmartLock root directories cannot be nested, either in enterprise or compliance mode, and hard links cannot cross SmartLock domain boundaries. Note that a SmartLock domain cannot be manually deleted. However, upon removal of a top level SmartLock directory, OneFS automatically deletes the associated SmartLock domain.
Once a file is WORM committed, or SmartLocked, it is indefinitely protected from modification or moves. When its expiry, or ‘committed until’ date is reached, the only actions that can be performed on a file are either deletion or extension of its ‘committed until’ date.
Enterprise mode permits storing data in enterprise directories in a non-rewriteable, non-erasable format, protecting data from deletion or modification, and enterprise directories in both enterprise and compliance modes. If a file in an enterprise directory is committed to a WORM state, it is protected from accidental deletion or modification until the retention period has expired. In Enterprise mode, you may also create regular directories, which are not subjected to SmartLock’s retention requirements. A cluster operating in enterprise mode provides the best of both worlds, providing WORM security, while retaining root access and full administrative control.
Any directory under /ifs is a potential SmartLock candidate, and it does not have to be empty before being designated as an enterprise directory. SmartLock and regular directories can happily coexist on the same cluster and once a directory is converted to a SmartLock directory, it is immediately ready to protect files that are placed there. SmartLock also automatically protects any subdirectories in a domain, and they inherit all the WORM settings of the parent directory – unless a specific exclusion is configured.
The following table indicates which type of files (and directories) can be created in each of the cluster modes:
| Directory | Enterprise mode | Compliance mode |
| Regular (non-WORM) directory | Y | Y |
| Enterprise directory (governed by system clock) | Y | Y |
| Compliance directory (governed by compliance clock) | N | N |
Both enterprise and compliance modes also permit the creation of non-WORM files and directories, which obviously are free from any retention requirements. Also, while an existing, empty enterprise directory can be upgraded to a compliance directory, it cannot be reverted back to an enterprise directory. Writes are permitted during a directory’s conversion to a SmartLock directory, but files cannot be committed until the transformation is complete.
| Attribute | Enterprise directory | Compliance directory |
| Customization file-retention dates | Y | Y |
| Post-commit write protection | Y | Y |
| SEC 17a-4(f) compliance | N | Y |
| Privileged delete | On | Off | Disabled | Disabled |
| Tamper-proof clock | N | Y |
| Root account | Y | N |
| Compadmin account | N | Y |
Regular users and apps are not permitted to move, delete, or change SmartLock-committed files. However, SmartLock includes the notion of a ‘privileged user’ with elevated rights (ie. root access) and the ability to delete WORM protected files. ‘Privileged deletes’ can only be performed on the cluster itself, not over the network, which adds an additional layer of control of privileged functions. The privileged user exists only in enterprise mode, and a privilege delete can also be performed by non-root users that have been assigned the ‘ISI_PRIV_IFS_WORM_DELETE’ RBAC role. The privileged delete capability is disabled by default, but can easily be enabled for enterprise directories (note that there is no privileged delete for compliance directories). It may also be permanently disabled to guard against deletion or modification from admin accounts.
Files in a SmartLock directory can be committed to a WORM state simply by removing their read-write privileges. A specific retention expiry date can be set on the file, which can be increased but not reduced. Any files that have been committed to a WORM state cannot be moved, even after their retention period has expired.
A file’s retention period expiration date can be set by modifying its access time (-atime), for example by using the CLI ‘touch’ command. However, note that simply accessing a file will not set the retention period expiration date.
If a file is ‘touched’ in a SmartLock directory without specifying a WORM release date, when committed, the retention period is automatically set to the default period specified for the SmartLock directory. If a default has not be set, the file is assigned a retention period of zero seconds. As such, the recommendation is clearly to specify a minimum retention period for all SmartLock directories.
After a directory has been marked as a SmartLock directory, any files committed to this directory are immutable until their retention time expires and cannot be delete, moved, or changed. The administrator sets retention dates on files, and can extend them but not shorten them. When a file’s retention policy expires, it becomes a normal file which can be managed or removed as desired.
Any uncommitted files in a SmartLock directory can be altered or moved at will, up until they are WORM committed, at which point they become immutable. Files can be committed to a SmartLock directory either locally via the cluster CLI, or from an NFS or SMB client.
The OneFS CLI chmod command can also be used to commit a file into the WORM state without removing the write permissions. This option alleviates the need for cluster admins to re-enable the permissions for users to modify the files after they have been released from WORM retention.
Be aware that, when using the cp -p command to copy a file with a read-only flag from the OneFS CLI to WORM domains, the target file will immediately be committed to WORM state. The read-only flag can be removed from source files before copying them to WORM domains if this is undesirable.
File retention times can be set in two ways: on a per-file basis, or using the directory default setting, with the file’s retention date overriding the directory default retention date. For any existing WORM files, the retention date can be extended but not reduced. Even after their retention date has expired, WORM-protected files cannot be altered while they reside in a SmartLock directory. Instead, they must be moved to a non-WORM directory before modification is permitted. Note that, in an enterprise SmartLock directory with ‘privileged delete’ enabled, a WORM state file can still be removed within its retention period.
A directory level ‘override retention date’ option is also available, which can be used automatically extend the retention date of files. Note that any files in the directory whose retention times were already set beyond the scope of the override are unaffected by an override.
OneFS 8.2 and later permits the exclusion of a directory inside a WORM domain from WORM retention policies and protection. Any content that is created later in the excluded directory is not SmartLock protected. 8.2 also introduced a ‘pending delete’ flag, which can be set on a compliance-mode WORM domain in order to delete the domain and the directories and files in it. Note that ‘pending delete’ cannot be configured on an enterprise-mode WORM domain.
Marking a domain for deletion is an irreversible action, after which no new files may be created, hard linked, or renamed into the domain. Existing files may not be committed or have their retention dates extended. Additionally, any SyncIQ replication tasks to and from the domain will fail.
In contrast to enterprise mode, and in order to maintain an elevated level of security and immutability, SmartLock compliance mode enforces some stringent administrative restrictions. Most notably, the root account (UID 0) loses its elevated privileges. Instead, clusters operating in compliance mode can use the ‘compadmin’ account to run some restricted commands with root privileges via ‘sudo’. These commands are specified in the /usr/local/etc/sudoers file.
Given the administrative restrictions of Compliance mode and its potential to affect both compliance data and enterprise data, it is strongly advised to only use compliance mode if mandated to do so under SEC rule 17a-4(f). In most cases, enterprise mode, with privileged delete disabled, offers a level of security that is more than adequate for the vast majority environments. Some fundamental differences between the two mode includes:
| Enterprise mode | Compliance mode |
| Governed by single system clock. | Governed by both system clock and compliance clock. |
| Data committed to WORM state only for specified retention period. WORM state file can have privileged delete capability within retention period. | Data written to compliance directories, when committed, can never be altered. |
| Root/adminstrator access retains full administrative control. | Root/administrator access is disabled. |
A directory cannot be upgraded to a SmartLock Compliance directory until the WORM compliance clock has been set on the cluster. This can be done from the CLI using the ‘isi worm cdate set’ syntax. Be aware that setting the compliance clock is a one-time deal, after which it cannot be altered.
The ComplianceStoreDelete job, introduced in OneFS 8.2, automatically tracks and removes expired files from the compliance store that were placed there as a result of SyncIQ conflict resolution. By default, this job is scheduled to run once per month at ‘low’ impact and priority level 6, but it can also be run manually on-demand.
If the decision is made to configure a cluster for Compliance mode, some tips to make the transition smoother include verifying that the cluster time is correct before putting the PowerScale cluster in Compliance mode. Plan on using RBAC for cluster access to perform administrative operations and data management, and be aware that for the CLI, ‘compadmin’ account represents a regular data user. For any root-owned data, perform all ownership or permission changes before upgrading to Compliance mode and avoid changing ownership of any system files. Review the permissions and ownership of any files that exclusively permit the root account to manage or write data to them. After upgrading to Compliance mode, if the OneFS configuration limits the relevant POSIX access permissions to specific directories or files, writing data or changing ownership of these objects will be blocked. Ensure that any SMB shares do not have ‘run-as-root’ configured before putting the PowerScale cluster into Compliance mode.
In the next article in this series, we’ll take a look at SmartLock’s configuration, compatibility, and use.