Received a couple of recent questions around the OneFS ChangelistCreate job, so thought it would make for a useful topic to explore over the course of the next couple of blog articles.
A changelist is essentially a catalog of the attributes and objects which changed between two checkpoints. For example, a list of the files, dirs, etc which were added, removed, modified, etc, typically between two file system snapshots. In OneFS, changelists are implemented as system B-trees (SBTs), with entries being indexed by ID or LIN, and containing information such as path, type and resulting size of the corresponding object. The SnapshotIQ framework is leveraged as the underlying mechanism for taking the required snapshots.
Historically, changelists were primarily utilized by SyncIQ as the foundation for differential replication. However, they are now used more widely, such as by FSAnalyze for InsightIQ, and IndexUpdate for the SmartPools FilePolicy job. Changelists are also of considerable interest to partners and vendors looking to integrate third party data protection and management software, solutions and tools with OneFS.
The OneFS Job Engine contains a class of jobs which utilize a ‘changelist’, rather than LIN-based scanning. The changelist approach analyzes two snapshots to find the LINs which changed, or delta, between the snapshots, and then examines and catalogs the detailed changes.
The ChangelistCreate job supports the creation of a changelist for any two snapshots with a common root path. The job itself is started manually and runs by default with a LOW impact policy and a priority of 5.
# isi job types view ChangelistCreate ID: ChangelistCreate Description: Create a list of changes between two snapshots with matching root paths. Enabled: Yes Policy: LOW Schedule: - Priority: 5
When a new ChangelistCreate job starts, it checks to see whether a finalized changelist with matching snapshot IDs already exists. If so, it completes quickly and successfully. Next, the job checks whether an in-progress changelist with matching snapshot IDs already exists. In this case, it retrieves and inspects the metadata entry (LIN 1) to find the ID of the job most recently operating on the changelist. If the ID references an active (e.g. paused) job, then the new job will exit with an EINPROGRESS error. Otherwise, the new job sets the metadata job ID value to its own job ID and proceeds with changelist creation.
ChangelistCreate scans all the snapshot tracking files (STFs) from the older snapshot (inclusive) to the newer snapshot (exclusive) in order to build a comprehensive list of applicable, changed LINs. When complete, it compiles a changelist through a combination of tree walks, stats, and path lookups.
Under the hood, the ChangelistCreate job has four distinct phases:
| Phase | Description |
| 1. Summarize | Performs basic validation and snapshot locking, determining any restart state, and summary STF creation. |
| 2. Examine | Handles the stat, path lookup, scoped tree walks, and changelist entry creation activity. |
| 3. Merge | Moves entries from a temporary ‘split-lin’ changelist to the primary changelist. |
| 4. Enumerate | Calculates the final changelist entry count. |
If a ChangelistCreate job fails or is cancelled before a changelist is finalized, the partial results can typically still be used by a subsequent job, with a couple of caveats. Specifically, stoppage at the summarize phase will result in the loss of all summary STF creation work, and stoppage at the ‘examine’ phase will result in any subsequent job(s) still iterating over all LINs in the summary STF but avoiding scoped tree walks, etc when a changelist entry already exists.
The ChangeListCreate ‘examine’ phase employs recursive logic, which is used to divide the task into work items to allow for distributed, interruptible processing.
For example, consider the processing of a LIN for a regular file, with no hardlinks or alternate data streams, which was changed at some point between the older and newer snapshots:
- The job engine calls a task’s ‘item_process’ routine, which, seeing that its work stack is empty, reads the next LIN from its allotted range in the summary STF, and then creates a work item for the LIN.
- This work item is pushed on to the work stack and a handler invoked.
- The handler runs, sees that the LIN references a file, sets the work item’s type approiately, and returns.
- The ‘item_process’ routine addresses the newest work item on the stack.
- The handler runs, a changelist entry is created for the LIN, and the handler pops the work item off the stack and returns.
- At this point the work stack is empty and the cycle repeats.
A depth-first approach ensures that a changelist entry is not created for a parent directory until entries have been created for all descendants covered by the recursive logic (the exception being LINs whose work items are split, which are written to a temporary “split-lin” changelist). As a result, if a prior job is interrupted, a subsequent job can easily identify and ignore branches that have already been fully processed.
From the administrator’s perspective, in addition to the regular job engine controls, OneFS also provides the ‘isi_changelist_mod’ CLI utility as the primary way to interact with changelists:
# isi_changelist_mod
Description:
Manage snapshot changelists.
Usage:
isi_changelist_mod -a cl_name ... Display all entries.
isi_changelist_mod -h Display help.
isi_changelist_mod -i {cl_name | --all} ... Describe changelist(s).
isi_changelist_mod -k {cl_name | --all} Kill changelist(s).
isi_changelist_mod -l List changelists.
Options for entry display (-a) and changelist describe (-i):
--B Replace non-printable path characters with octal codes.
See ascii(7).
--b As --B, but use C escape codes whenever possible,
e.g. \t for TAB.
--p Only display path.
--q Replace non-printable path characters with '?'.
--s Append '/' to paths of directories.
--t Use shquote(3) on path strings to make them suitable for
command-line arguments.
--w Display raw path strings. (Default.)
Additional options for entry display (-a):
--d den Fractional entry range denominator (2 <= den <= 1024).
--n num Fractional entry range numerator (1 <= num <= den).
--v Prepend '+' and '-' to paths of added and removed entries.
For changelist entry st_* field descriptions, see stat(2).
Here’s an example command output entry for a particular changed file:
st_ino=4297261065 st_mode=0100644 st_size=3 st_atime=1429572712 st_mtime=1429572712 st_ctime=1429572712 st_flags=224 cl_flags=01 path=/ifs/data/test1/f3
The ‘st_*’ fields above are derived from ‘stat’ output, and correspond to the following:
| ST Field | Description |
| st_ino | File’s inode number. |
| st_mode | File type and mode . |
| st_size | Size of the file (if it is a regular file or symbolic link). |
| st_atime | Last access time of file data (epoch). |
| st_mtime | Time of last modification of file data (epoch). |
| st_ctime | File’s last status change timestamp (epoch time of last change to the inode) |
| st_flags | User defined flags enabled for the file. |
Similarly, the changelist entry status information field ‘cl_flags’ can have one of the following values:
| CL Field | Description |
| 01 | Added or moved to. |
| 02 | Removed of moved from. |
| 04 | Path changed – moved to/from. |
| 10 | Contains Alternate Data Stream(s) |
| 20 | Is an Alternate Data Stream |
| 40 | Hardlinks exist. |
So, from the example above, we can see that the object has the following pertinent attributes:
- Its inode number is 4297261065
- It resides in the file system at /ifs/data/test1/f3.
- The st_mode=0100644 shows it’s a file (value=1)
- with user/group/all mode bits indicating user=read/write (6), group=read (4), everyone=read(4).
- It’s newly created file, as indicated by the cl_flags=01 field’
If desired, OneFS can also be allow multiple ChangeListCreate jobs to run concurrently:
# isi_gconfig -t job-config jobs.types.changelistcreate.allow_multiple_instances=trueIn the next article, we’ll walk through an example of creating, accessing and managing a changelist via both the CLI and platform API.