How to connect to HBase from spark using Kerberos
0
Can anyone please tell me how to connect to HBase from spark using kerberos. Am using the below code to create connection to HBase but its still having issues.
val genericMessage = messages.mapPartitions(iter => {
val context = TaskContext.get
logger.info((s"log - Process for partition: ${context.partitionId} "))
val partitionId: Int = context.partitionId
val conf: Configuration = HBaseConfiguration.create()
conf.set("hbase.zookeeper.quorum", ZOOKEEPER_QOURUM)
conf.set("hbase.rpc.timeout", "1800000")
conf.set("hbase.client.scanner.timeout.period", "1800000")
conf.set("hadoop.security.authentication", "kerberos")
import org.apache.hadoop.security.UserGroupInformation
UserGroupInformation.loginUserFromKeytab("customer@mail.com", keyTab)
val connection: Connection = ConnectionFactory.createConnection(conf)
val custTable Table = connection.getTable(TableName.valueOf("prod:cusotmer"))
val avroMessages = iter.map(msg => (msg._1, enrichment(rec._1, decodeBinaryToAvro(rec._2)))
connection.close()
custTable.close()
avroMessages
})
Thanks
apache-spark hbase spark-streaming
asked Nov 14 '18 at 23:31
IndiraIndira
1415
|
show 4 more comments
0
Can anyone please tell me how to connect to HBase from spark using kerberos. Am using the below code to create connection to HBase but its still having issues.
val genericMessage = messages.mapPartitions(iter => {
val context = TaskContext.get
logger.info((s"log - Process for partition: ${context.partitionId} "))
val partitionId: Int = context.partitionId
val conf: Configuration = HBaseConfiguration.create()
conf.set("hbase.zookeeper.quorum", ZOOKEEPER_QOURUM)
conf.set("hbase.rpc.timeout", "1800000")
conf.set("hbase.client.scanner.timeout.period", "1800000")
conf.set("hadoop.security.authentication", "kerberos")
import org.apache.hadoop.security.UserGroupInformation
UserGroupInformation.loginUserFromKeytab("customer@mail.com", keyTab)
val connection: Connection = ConnectionFactory.createConnection(conf)
val custTable Table = connection.getTable(TableName.valueOf("prod:cusotmer"))
val avroMessages = iter.map(msg => (msg._1, enrichment(rec._1, decodeBinaryToAvro(rec._2)))
connection.close()
custTable.close()
avroMessages
})
Thanks
apache-spark hbase spark-streaming
asked Nov 14 '18 at 23:31
IndiraIndira
1415
Spark executors have to connect individually to HBase, but they don't have a Kerberos ticket (except inlocalmode). Is that the kind of "issue" you are talking about??
– Samson Scharfrichter
Nov 15 '18 at 18:37
Also, RTFM > hbase.apache.org/book.html#spark > that HBase module (contributed by Cloudera) handles Kerberos out of the box.
– Samson Scharfrichter
Nov 15 '18 at 18:42
@SamsonScharfrichter, thanks for the reply. I tried to use --principal and --keytab options but still it didnt work. Its throwing NullPointer Exception when I use these options. And if I use the below in code then its throwing Unable to obtain password from user. System.setProperty("java.security.krb5.conf", "/etc/krb5.conf") UserGroupInformation.loginUserFromKeytab("customer@CREALM.NET", keyTab)
– Indira
Nov 16 '18 at 6:59
Don't try to manage Kerberos authentication inside executor code. The proper way is to let the Spark init obtain "tokens" for HDFS, Hive, HBase etc then broadcast the "tokens" to each executor; the trick is token-based authentication in HBase is not documented... so use the Spark-HBase connector instead, withHBaseContext, unless you are ready to read the Spark and HBase code on GitHub and also have several years of experience with Kerberos and the scars to prove it.
– Samson Scharfrichter
Nov 16 '18 at 17:08
Recommended reading: stackoverflow.com/questions/44265562/…
– Samson Scharfrichter
Nov 16 '18 at 17:18
|
show 4 more comments
0
0
0
Can anyone please tell me how to connect to HBase from spark using kerberos. Am using the below code to create connection to HBase but its still having issues.
val genericMessage = messages.mapPartitions(iter => {
val context = TaskContext.get
logger.info((s"log - Process for partition: ${context.partitionId} "))
val partitionId: Int = context.partitionId
val conf: Configuration = HBaseConfiguration.create()
conf.set("hbase.zookeeper.quorum", ZOOKEEPER_QOURUM)
conf.set("hbase.rpc.timeout", "1800000")
conf.set("hbase.client.scanner.timeout.period", "1800000")
conf.set("hadoop.security.authentication", "kerberos")
import org.apache.hadoop.security.UserGroupInformation
UserGroupInformation.loginUserFromKeytab("customer@mail.com", keyTab)
val connection: Connection = ConnectionFactory.createConnection(conf)
val custTable Table = connection.getTable(TableName.valueOf("prod:cusotmer"))
val avroMessages = iter.map(msg => (msg._1, enrichment(rec._1, decodeBinaryToAvro(rec._2)))
connection.close()
custTable.close()
avroMessages
})
Thanks
apache-spark hbase spark-streaming
asked Nov 14 '18 at 23:31
IndiraIndira
1415
Can anyone please tell me how to connect to HBase from spark using kerberos. Am using the below code to create connection to HBase but its still having issues.
val genericMessage = messages.mapPartitions(iter => {
val context = TaskContext.get
logger.info((s"log - Process for partition: ${context.partitionId} "))
val partitionId: Int = context.partitionId
val conf: Configuration = HBaseConfiguration.create()
conf.set("hbase.zookeeper.quorum", ZOOKEEPER_QOURUM)
conf.set("hbase.rpc.timeout", "1800000")
conf.set("hbase.client.scanner.timeout.period", "1800000")
conf.set("hadoop.security.authentication", "kerberos")
import org.apache.hadoop.security.UserGroupInformation
UserGroupInformation.loginUserFromKeytab("customer@mail.com", keyTab)
val connection: Connection = ConnectionFactory.createConnection(conf)
val custTable Table = connection.getTable(TableName.valueOf("prod:cusotmer"))
val avroMessages = iter.map(msg => (msg._1, enrichment(rec._1, decodeBinaryToAvro(rec._2)))
connection.close()
custTable.close()
avroMessages
})
Thanks
apache-spark hbase spark-streaming
apache-spark hbase spark-streaming
asked Nov 14 '18 at 23:31
IndiraIndira
1415
asked Nov 14 '18 at 23:31
IndiraIndira
1415
asked Nov 14 '18 at 23:31
IndiraIndira
1415
asked Nov 14 '18 at 23:31
IndiraIndira
1415
asked Nov 14 '18 at 23:31
IndiraIndira
1415
1415
Spark executors have to connect individually to HBase, but they don't have a Kerberos ticket (except inlocalmode). Is that the kind of "issue" you are talking about??
– Samson Scharfrichter
Nov 15 '18 at 18:37
Also, RTFM > hbase.apache.org/book.html#spark > that HBase module (contributed by Cloudera) handles Kerberos out of the box.
– Samson Scharfrichter
Nov 15 '18 at 18:42
@SamsonScharfrichter, thanks for the reply. I tried to use --principal and --keytab options but still it didnt work. Its throwing NullPointer Exception when I use these options. And if I use the below in code then its throwing Unable to obtain password from user. System.setProperty("java.security.krb5.conf", "/etc/krb5.conf") UserGroupInformation.loginUserFromKeytab("customer@CREALM.NET", keyTab)
– Indira
Nov 16 '18 at 6:59
Don't try to manage Kerberos authentication inside executor code. The proper way is to let the Spark init obtain "tokens" for HDFS, Hive, HBase etc then broadcast the "tokens" to each executor; the trick is token-based authentication in HBase is not documented... so use the Spark-HBase connector instead, withHBaseContext, unless you are ready to read the Spark and HBase code on GitHub and also have several years of experience with Kerberos and the scars to prove it.
– Samson Scharfrichter
Nov 16 '18 at 17:08
Recommended reading: stackoverflow.com/questions/44265562/…
– Samson Scharfrichter
Nov 16 '18 at 17:18
|
show 4 more comments
Spark executors have to connect individually to HBase, but they don't have a Kerberos ticket (except inlocalmode). Is that the kind of "issue" you are talking about??
– Samson Scharfrichter
Nov 15 '18 at 18:37
Also, RTFM > hbase.apache.org/book.html#spark > that HBase module (contributed by Cloudera) handles Kerberos out of the box.
– Samson Scharfrichter
Nov 15 '18 at 18:42
@SamsonScharfrichter, thanks for the reply. I tried to use --principal and --keytab options but still it didnt work. Its throwing NullPointer Exception when I use these options. And if I use the below in code then its throwing Unable to obtain password from user. System.setProperty("java.security.krb5.conf", "/etc/krb5.conf") UserGroupInformation.loginUserFromKeytab("customer@CREALM.NET", keyTab)
– Indira
Nov 16 '18 at 6:59
Don't try to manage Kerberos authentication inside executor code. The proper way is to let the Spark init obtain "tokens" for HDFS, Hive, HBase etc then broadcast the "tokens" to each executor; the trick is token-based authentication in HBase is not documented... so use the Spark-HBase connector instead, withHBaseContext, unless you are ready to read the Spark and HBase code on GitHub and also have several years of experience with Kerberos and the scars to prove it.
– Samson Scharfrichter
Nov 16 '18 at 17:08
Recommended reading: stackoverflow.com/questions/44265562/…
– Samson Scharfrichter
Nov 16 '18 at 17:18
Spark executors have to connect individually to HBase, but they don't have a Kerberos ticket (except in
local mode). Is that the kind of "issue" you are talking about??– Samson Scharfrichter
Nov 15 '18 at 18:37
Spark executors have to connect individually to HBase, but they don't have a Kerberos ticket (except in
local mode). Is that the kind of "issue" you are talking about??– Samson Scharfrichter
Nov 15 '18 at 18:37
Also, RTFM > hbase.apache.org/book.html#spark > that HBase module (contributed by Cloudera) handles Kerberos out of the box.
– Samson Scharfrichter
Nov 15 '18 at 18:42
Also, RTFM > hbase.apache.org/book.html#spark > that HBase module (contributed by Cloudera) handles Kerberos out of the box.
– Samson Scharfrichter
Nov 15 '18 at 18:42
@SamsonScharfrichter, thanks for the reply. I tried to use --principal and --keytab options but still it didnt work. Its throwing NullPointer Exception when I use these options. And if I use the below in code then its throwing Unable to obtain password from user. System.setProperty("java.security.krb5.conf", "/etc/krb5.conf") UserGroupInformation.loginUserFromKeytab("customer@CREALM.NET", keyTab)
– Indira
Nov 16 '18 at 6:59
@SamsonScharfrichter, thanks for the reply. I tried to use --principal and --keytab options but still it didnt work. Its throwing NullPointer Exception when I use these options. And if I use the below in code then its throwing Unable to obtain password from user. System.setProperty("java.security.krb5.conf", "/etc/krb5.conf") UserGroupInformation.loginUserFromKeytab("customer@CREALM.NET", keyTab)
– Indira
Nov 16 '18 at 6:59
Don't try to manage Kerberos authentication inside executor code. The proper way is to let the Spark init obtain "tokens" for HDFS, Hive, HBase etc then broadcast the "tokens" to each executor; the trick is token-based authentication in HBase is not documented... so use the Spark-HBase connector instead, with
HBaseContext, unless you are ready to read the Spark and HBase code on GitHub and also have several years of experience with Kerberos and the scars to prove it.– Samson Scharfrichter
Nov 16 '18 at 17:08
Don't try to manage Kerberos authentication inside executor code. The proper way is to let the Spark init obtain "tokens" for HDFS, Hive, HBase etc then broadcast the "tokens" to each executor; the trick is token-based authentication in HBase is not documented... so use the Spark-HBase connector instead, with
HBaseContext, unless you are ready to read the Spark and HBase code on GitHub and also have several years of experience with Kerberos and the scars to prove it.– Samson Scharfrichter
Nov 16 '18 at 17:08
Recommended reading: stackoverflow.com/questions/44265562/…
– Samson Scharfrichter
Nov 16 '18 at 17:18
Recommended reading: stackoverflow.com/questions/44265562/…
– Samson Scharfrichter
Nov 16 '18 at 17:18
|
show 4 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53310324%2fhow-to-connect-to-hbase-from-spark-using-kerberos%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53310324%2fhow-to-connect-to-hbase-from-spark-using-kerberos%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Spark executors have to connect individually to HBase, but they don't have a Kerberos ticket (except in
localmode). Is that the kind of "issue" you are talking about??– Samson Scharfrichter
Nov 15 '18 at 18:37
Also, RTFM > hbase.apache.org/book.html#spark > that HBase module (contributed by Cloudera) handles Kerberos out of the box.
– Samson Scharfrichter
Nov 15 '18 at 18:42
@SamsonScharfrichter, thanks for the reply. I tried to use --principal and --keytab options but still it didnt work. Its throwing NullPointer Exception when I use these options. And if I use the below in code then its throwing Unable to obtain password from user. System.setProperty("java.security.krb5.conf", "/etc/krb5.conf") UserGroupInformation.loginUserFromKeytab("customer@CREALM.NET", keyTab)
– Indira
Nov 16 '18 at 6:59
Don't try to manage Kerberos authentication inside executor code. The proper way is to let the Spark init obtain "tokens" for HDFS, Hive, HBase etc then broadcast the "tokens" to each executor; the trick is token-based authentication in HBase is not documented... so use the Spark-HBase connector instead, with
HBaseContext, unless you are ready to read the Spark and HBase code on GitHub and also have several years of experience with Kerberos and the scars to prove it.– Samson Scharfrichter
Nov 16 '18 at 17:08
Recommended reading: stackoverflow.com/questions/44265562/…
– Samson Scharfrichter
Nov 16 '18 at 17:18