DEV Community: Shaine Ismail

Windows GIT certificate

Shaine Ismail — Wed, 27 May 2020 16:10:16 +0000

If you are running git on windows many things can go wrong, one of the more irritating problem are with certificates.

to use the windows certificate store set the following

git config --system http.sslbackend schannel

From the release notes

It is now possible to switch between Secure Channel and OpenSSL for Git's HTTPS transport by setting the http.sslBackend config variable to "openssl" or "schannel".

Understanding Hadoop.proxy users

Shaine Ismail — Thu, 21 May 2020 12:55:18 +0000

When you have a secure Hadoop cluster you need the ability of services to be able to authenticate and execute applications on behalf of the user.

If a service user with name service_runner wants submit a yarn job and access HDFS for user bob. The service_runner user has kerberos credentials but bob does not, but the service_runner does not have the level of access the user bob does.

It is required that user bob is used to connect to the namenode or the job tracker on a connection authenticated with service_runner kerberos credentials. In other words service_runner needs to impersonate user bob.

Host level control

   <property>
     <name>hadoop.proxyuser.service_runner.hosts</name>
     <value>10.222.0.0/16,10.113.221.221</value>
   </property>
   <property>
     <name>hadoop.proxyuser.service_runner.users</name>
     <value>bob</value>
   </property>

service_runner can impersonate bob from 10.222.0.0-15 and 10.113.221.221

group level control too open

   <property>
     <name>hadoop.proxyuser.service_runner.hosts</name>
     <value>*</value>
   </property>
   <property>
     <name>hadoop.proxyuser.service_runner.users</name>
     <value>*</value>
   </property>

service_runner can impersonate any user on the cluster from any host

group level control close

   <property>
     <name>hadoop.proxyuser.service_runner.hosts</name>
     <value>*</value>
   </property>
   <property>
     <name>hadoop.proxyuser.service_runner.groups</name>
     <value>service_runner_execute</value>
   </property>

service_runner can impersonate any user that is in the service_runner_execute group

So when you are on boarding a new product to your Hadoop cluster adding the recommended setting might open your cluster to security issues, especially to malicious insiders. Consider limiting to a known group of users and having your security explicit inclusion rather than open to all by default.

link to the docs

Calling a stored Procedure SQL Server stored procedure from Spark

Shaine Ismail — Fri, 15 May 2020 14:21:13 +0000

Not really a regular thing people need to do and there are options to insert the record set into a temp table which means that you can go directly into data frame. But that is an option that you need your DBA's to switch on.

the following uses a jdbc connection and a result set into a RDD then into a dataframe

import java.sql.DriverManager
import java.sql.Connection

import org.apache.spark.sql.types.{StructType, StructField, StringType,IntegerType};
import java.sql.ResultSet

//driver = 'com.microsoft.sqlserver.jdbc.SQLServerDriver' 

val username = ""
val pass = ""
val url = "jdbc:sqlserver://<host>:<port>;databaseName=<db>"



val columns = Seq ("DateCreated", "Action", "ServerName", "IPAddress", "Domain")

val schema = StructType(List(
    StructField("DateCreated", StringType, nullable = true),
    StructField("Action", StringType,  nullable = true),
    StructField("ServerName", StringType,  nullable = true),
    StructField("IPAddress", StringType,  nullable = true),
    StructField("Domain", StringType,  nullable = true)
))


def parseResultSet(rs: ResultSet): Row = {
    val resultSetRecord = columns.map(c => rs.getString(c))
    Row(resultSetRecord:_*)
}

def resultSetToIter(rs: ResultSet)(f: ResultSet => Row): Iterator[Row] = new Iterator[Row] {
    def hasNext: Boolean = rs.next()
    def next(): Row = f(rs)
}

def paralleliseResultSet(rs: ResultSet, spark:SparkSession): DataFrame = {
  val rdd = spark.sparkContext.parallelize(resultSetToIter(rs(parseResultSet).toSeq)     
  spark.createDataFrame(rdd, schema)
}

val conn = DriverManager.getConnection(url, username, pass)
val rs = conn.createStatement.executeQuery("exec <sp name>")
val df: DataFrame = paralleliseResultSet(rs,spark)
df.createOrReplaceTempView("df")

spark.sql("""select * from df""").show(10, False)