The new OneFS 9.3 sees some useful features added to its S3 object protocol stack, including:
- Chunked Upload
- Delete Multiple Objects support
- Non-slash delimiter support for ListObjects/ListObjectsV2
When uploading data to OneFS via S3, there are two types of uploading options for authenticating requests using the S3 Authorization header:
- Transfer payload in a single chunk
- Transfer payload in multiple chunks (chunked upload)
Applications that typically use the chunked upload option by default include Restic, Flink, Datadobi, and the AWS S3 Java SDK. The new 9.3 release enables these and other applications to work seamlessly with OneFS.
Chunked upload, as the name suggests, facilitates breaking data payload into smaller units, or chunks for more efficient upload. These can be fixed or variable-size, and chunking aids performance by avoiding reading the entire payload in order to calculate the signature. Instead, for the first chunk, a seed signature is calculated which uses only the request headers. The second chunk contains the signature for the first chunk, and each subsequent chunk contains the signature for the preceding one. At the end of the upload, a zero byte chunk is transmitted which contains the last chunk’s signature. This protocol feature is described in more detail in the AWS S3 Chunked Upload documentation.
The AWS S3 DeleteObjects API enables the deletion of multiple objects from a bucket using a single HTTP request. If you know the object keys that you wish to delete, the DeleteObjects API provides an efficient alternative to sending individual delete requests, reducing per-request overhead.
For example, the following python code can be used to delete the three objects file1, file2, and file3 from bkt01 in a single operation:
import boto3
# set HOST IP, user access id and secret key
HOST='192.168.198.10' # Your SmartConnect name or cluster IP goes here
USERNAME='1_s3test_accid' # Your access ID
USERKEY='WttVbuRv60AXHiVzcYn3b8yZBtKc' # Your secret key
URL = 'http://{}:9020'.format(HOST)
s3 = boto3.resource('s3')
session = boto3.Session()
s3client = session.client(service_name='s3',aws_access_key_id=USERNAME,aws_secret_access_key=USERKEY,endpoint_url=URL,use_ssl=False,verify=False)
bkt_name='bkt01'
response=s3client.delete_objects(
Bucket='bkt01',
Delete={
'Objects': [
{
'Key': 'file1'
},
{
'Key': 'file2'
},
{
'Key': 'file3'
}
]
}
)
print(response)
Note that Boto3, the AWS S3 SDK for python, is used in the code above. Boto3 can be downloaded here and installed on a Linux client via pip (ie. # pip install boto3).
Another S3 feature that’s added in OneFS 9.3 is non-slash delimiter support. The AWS S3 data model is a flat structure with no physical hierarchy of directories or folders: A bucket is created, under which objects are stored. However, AWS S3 does make provision for a logical hierarchy using object key name prefixes and delimiters to support a rudimentary concept of folders, as described in Amazon S3 Delimiter and Prefix. In prior OneFS releases, only a slash (‘/’) was supported as a delimiter. However, the new OneFS 9.3 release now expands support to include non-slash delimiters for listing objects in buckets. Also, the new delimiter can comprise multiple characters.
To illustrate this, take the keys “a/b/c”, “a/bc/e” , abc”:
- If the delimiter is “b” with no prefix, “a/b” and “ab” are returned as the common prefix.
- With delimiter “b” and prefix “a/b”, “a/b/c” and “a/bc/e” will be returned.
The delimiter can also have either ‘no slash’ or ‘slash’ at the end. For example, “abc”, “/”, “xyz/” are all supported. However, “a/b”, “/abc”, “//” are invalid.
In the following example, three objects (file1, file2, and file3) are uploaded from a Linux client to a cluster via the OneFS S3 protocol with object keys, and stored under the following topology:
# tree bkt1 bkt1 ├── dir1 │ ├── file2 │ └── sub-dir1 │ └── file3 └── file1 2 directories, 3 files
These objects can be listed using ‘sub’ as the delimiter value by running the following python code:
import boto3
# set HOST IP, user access id and secret key
HOST='192.168.198.10' # Your SmartConnect name or cluster IP goes here
USERNAME='1_s3test_accid' # Your access ID
USERKEY=' WttVbuRv60AXHiVzcYn3b8yZBtKc' # Your secret key
URL = 'http://{}:9020'.format(HOST)
s3 = boto3.resource('s3')
session = boto3.Session()
s3client = session.client(service_name='s3',aws_access_key_id=USERNAME,aws_secret_access_key=USERKEY,endpoint_url=URL,use_ssl=False,verify=False)
bkt_name='bkt1'
response=s3client.list_objects(
Bucket=bkt_name,
Delimiter='sub'
)
print(response)
The keys ‘file1’ and ‘dir1/file2’ are returned in the , and ‘dir1/sub’ is returned as a common prefix.
{'ResponseMetadata': {'RequestId': '564950507', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'connection': 'keep-alive', 'x-amz-request-id': '564950507', 'content-length': '796'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Marker': '', 'Contents': [{'Key': 'dir1/file2', 'LastModified': datetime.datetime(2021, 11, 24, 16, 15, 6, tzinfo=tzutc()), 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'Size': 0, 'StorageClass': 'STANDARD', 'Owner': {'DisplayName': 's3test', 'ID': 's3test'}}, {'Key': 'file1', 'LastModified': datetime.datetime(2021, 11, 24, 16, 10, 43, tzinfo=tzutc()), 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'Size': 0, 'StorageClass': 'STANDARD', 'Owner': {'DisplayName': 's3test', 'ID': 's3test'}}], 'Name': 'bkt1', 'Prefix': '', 'Delimiter': 'sub', 'MaxKeys': 1000, 'CommonPrefixes': [{'Prefix': 'dir1/sub'}]}
OneFS 9.3 also delivers significant improvements to the S3 multi-part upload functionality. In prior OneFS versions, each constituent piece of an upload was written to a separate file, and all the parts concatenated on a completion request. As such, the concatenation process could take a significant duration for large file.
With the new OneFS 9.3 release, multi-part upload instead writes data directly into a single file, so completion is near-instant. The multiple parts are consecutively numbered, and all have same size except for the final one. Since no re-upload or concatenation is required, the process is both lower overhead as well as significantly quicker.
OneFS 9.3 also includes improved handling of inter-level directories. For example, if ‘a/b’ is put on a cluster via S3, the directory ‘a’ is created implicitly. In previous releases, if ‘b’ was then deleted, the directory ‘a’ remained and was treated as an object. However, with OneFS 9.3, the directory is still created and left, but is now identified as an inter-level directory. As such, it is not shown as an object via either ‘Get Bucket’ or ‘Get Object’. With 9.3, an S3 client can now remove a bucket if it only has inter-level directories. In prior releases, this would have failed with a ‘bucket not empty’ error. However, the multi-protocol behavior is unchanged, so a directory created via another OneFS protocol, such as NFS, is still treated as an object. Similarly, if an inter-level directory was created on a cluster prior to a OneFS 9.3 upgrade, that directory will continue to be treated as an object.