By Boni Bruno, Chief Solutions Architect | Dell EMC
Dell EMC Isilon
Dell EMC Isilon solves the hard scaling problems our customers have with consolidating and storing large amounts of unstructured data. Isilon’s scale-out design and multi-protocol support provides efficient deployment of data lakes as well as support for big data platforms such as Hadoop, Spark, and Kafka to name a few examples.
In fact, the embedded HDFS implementation that comes with Isilon OneFS has been CERTIFIED by Cloudera for both HDP and CDH Hadoop distributions. Dell EMC has also been recognized by Gartner as a Leader in the Gartner Magic Quadrant for Distributed File Systems and Object Storage four years in a row. To that end, Dell EMC is delighted to announce that Isilon is a validated HDFS tiering solution for Microsoft’s SQL Server Big Data Clusters.
SQL Server Big Data Clusters & HDFS Tiering with Dell EMC Isilon
SQL Server Big Data Clusters allow you to deploy clusters of SQL Server, Spark, and HDFS containers on Kubernetes. With these components, you can combine and analyze MS SQL relational data with high-volume unstructured data on Dell EMC Isilon. This means that Dell EMC customers who have data on their Isilon clusters can now make their data available to their SQL Server Big Data Clusters for analytics using the embedded HDFS interface that comes with Isilon OneFS.
Note: The HDFS Tiering feature of SQL Server 2019 Big Data Clusters currently does not support Cloudera Hadoop, Isilon provides immediate access to HDFS data with or without a Hadoop distribution being deployed in the customers’ environment. This is a unique value proposition of Dell EMC Isilon storage solution for SQL Server Big Data Clusters. Unstructured data stored on Isilon is directly accessed over HDFS and will transparently appear as local data to the SQL Server Big Data Cluster platform.
The Figure below depicts the overall architecture between SQL Server Big Data Cluster platform and Dell EMC Isilon or ECS storage solutions.
Dell EMC provides two storage solutions that can integrate with SQL Server Big Data Clusters. Dell EMC Isilon provides a high-performance scale-out HDFS solution and Dell EMC ECS provides a high-capacity scale-out S3A solution, both are on-premise storage solutions.
We are currently working with the Microsoft’s Azure team to get these storage solutions available to customers in the cloud as well. The remainder of this article provides details on how Dell EMC Isilon integrates with SQL Server Big Data Cluster over HDFS.
Setting up HDFS on Dell EMC Isilon
Enabling HDFS on Isilon is as simple as clicking a button in the OneFS GUI. Customers have the choice of having multiple access zones if needed, access zones provide a logical separation of the data and users with support for independent role-based access controls. For the purposes of this article, a “msbdc” access zone will be used for reference. By default, HDFS is disabled on a given access zone as shown below:
To activate HDFS, simply click the Activate HDFS button. Note: HDFS licenses are free with the purchase of Isilon, HDFS licenses can be installed under Cluster Management\Licenses.
Once an HDFS license in installed and HDFS is activated on a given access zone, the HDFS settings can be viewed as shown below:
The GUI allows you to easily change the HDFS block size, Authentication Type, Enable the Ranger Security Plugin, etc. Isilon OneFS also supports various authentication providers and additional protocols as shown below:
Simply pick the authentication provider of your choice and specify the provider details to enable remote authentication services on Isilon. Note: Isilon OneFS has a robust security architecture and authentication, identity management, and authorization stack, you can find more details here.
The multi-protocol support included with Isilon allows customers to land data on Isilon over SMB, NFS, FTP, or HTTP and make all or part of the data available to SQL Server Big Data Clusters over HDFS without having a Hadoop cluster installed – Beautiful!
A key performance aspect of Dell EMC Isilon is the scale-out design of both the hardware and the integrated OneFS storage operating system. Isilon OneFS provides a unique SmartConnect feature that provides HDFS namenode and datanode load balancing and redundancy.
To use SmartConnect, simply delegate a sub-domain of your choice on your internal DNS server to Isilon and OneFS will automatically load balance all the associated HDFS connections from SQL Server Big Data Clusters transparently across all the physical nodes on the Isilon storage cluster.
The SmartConnect zone name is configured under Cluster Management\Network Configuration\ in the OneFS GUI as shown below:
In the example screen shot above, the SmartConnect Zone name is msbdc.dellemc.com, this means the delegated subdomain on the internal DNS server should be msbdc, a nameserver record for this msbdc subdomain needs to point to the defined SmartConnect Service IP.
The Service IP information is in the subnet details in the OneFS GUI as shown below:
In the above example, the service IP address is 10.10.10.10. So, creating DNS records for 10.10.10.10 (e.g. isilon.dellemc.com) and a NS record for msbdc.dellemc.com that is served by isilon.dellemc.com (10.10.10.10) is all that would be needed on the internal DNS server configuration to take advantage of the built-in load balancing capabilities of Isilon.
Use “ping” to validate the SmartConnect/DNS configuration. Multiple ping tests to msbdc.dellemc.com should result with different IP address responses returned by Isilon, the range of IP addresses returned is defined by the IP Pool Range in the Isilon GUI.
SQL Server Big Data Cluster would simply have a single mount configuration pointing to the defined SmartConnect Zone name on Isilon. Details on how to setup the HDFS mount to Isilon from SQL Server Big Data Cluster is presented in the next section.
SmartConnect makes storage administration easy. If more storage capacity is required, simply add more Isilon nodes to the cluster and storage capacity and I/O performance instantly increases without having to make a single configuration change to the SQL Server Big Data Clusters – BRILLIANT!
With HDFS enabled, the access zone defined, and the network/DNS configuration complete, the Isilon storage system can now be mounted by SQL Server Big Data Clusters.
Mounting Dell EMC Isilon from SQL Server Big Data Cluster
Assuming you have a SQL Server Big Data Cluster running, begin with opening a terminal session to connect to your SQL Server Big Data Cluster. You can obtain the IP address of the end point controller-svc-external service of your cluster with the following command:
Using the IP of the controller end point obtained from the above command, log into your big data cluster:
Mount Isilon using HDFS on your SQL Server Big Data Cluster with the following command:
Note: hdfs://msbdc.dellemc.com is shown as an example, the hdfs uri must match the SmartConnect Zone name defined in the Isilon configuration. The data directory specified is also an example, any directory name that exists within the Isilon Access Zone can be used. Also, the mount point /mount1 that is shown above is just an example, any name can be used for the mount point.
An example of a successful response of the above mount command is shown below:
Create mount /mount1 submitted successfully. Check mount status for progress.
Check the mount status with the following command:
sample output:
Run an hdfs shell and list the contents on Isilon:
sample output:
In addition to using hdfs shell commands, you can use tools like Azure Data Studio to access and browse files over the HDFS service on Dell EMC Isilon. The example below is using Spark to read the data over HDFS:
To learn more about Dell EMC Isilon, please visit us at DellEMC.com.