从创建的Glue客户端中提取AWS Glue凭据Scala。

huangapple go评论85阅读模式
英文:

Extract AWS Glue credentials from created glue client Scala

问题

我正在尝试提取嵌入式连接凭据,参考这个问题
但是我遇到以下错误。

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._
import org.apache.spark.sql.expressions.{Window => W}
import org.apache.spark.sql.{functions => F}
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.GlueArgParser

import com.amazonaws.services.glue.DynamicFrame

import com.amazonaws.regions.Regions
import com.amazonaws.services.glue.model._

import com.amazonaws.services.glue.{AWSGlue, AWSGlueClient}
import scala.collection.JavaConverters.{mapAsJavaMapConverter, seqAsJavaListConverter}
import com.amazonaws.services.sagemaker.sparksdk.IAMRole

val sc = spark.sparkContext
val glueContext: GlueContext = new GlueContext(sc)

val region = Regions.fromName("us-east-1")

// 创建AWS Glue客户端的函数
def glueClient(region: Regions): AWSGlue = AWSGlueClient.builder().withRegion(region).build()

val glue = glueClient(region = region)

glue.getConnection("{Name: name-of-embedded-connection,HidePassword: False}")

遇到错误: :71: 错误: 类型不匹配; 找到: String("{Name: preprod-samtec-redw,HidePassword: False}") 需要: com.amazonaws.services.glue.model.GetConnectionRequest
glue.getConnection("{Name: name-of-embedded-connection,HidePassword: False}")

英文:

I am attempting to extract emmbedded connection credentials in reference to this question.
But I get the following error.

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._  //import everything from the package    
import org.apache.spark.sql.expressions.{Window => W} 
import org.apache.spark.sql.{functions => F}
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.GlueArgParser 

import com.amazonaws.services.glue.DynamicFrame

import com.amazonaws.regions.Regions
import com.amazonaws.services.glue.model._

import com.amazonaws.services.glue.{AWSGlue, AWSGlueClient}
import scala.collection.JavaConverters.{mapAsJavaMapConverter, seqAsJavaListConverter}
import com.amazonaws.services.sagemaker.sparksdk.IAMRole

val sc = spark.sparkContext
val glueContext: GlueContext = new GlueContext(sc)

val region = Regions.fromName("us-east-1")

// Function to create AWS glue client
def glueClient(region: Regions):
AWSGlue = AWSGlueClient.builder().withRegion(region).build()

val glue = glueClient(region =region)

glue.getConnection("{Name: name-of-embedded-connection,HidePassword: False}")

> > An error was encountered: <console>:71: error: type mismatch; found : String("{Name: preprod-samtec-redw,HidePassword: False}") required:
> com.amazonaws.services.glue.model.GetConnectionRequest
> glue.getConnection("{Name: name-of-embedded-connection,HidePassword: False}")

答案1

得分: 1

正如错误提示所指出的,您需要提供一个GetConnectionRequest对象给getConnection方法,类似于:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._  // 从包中导入所有内容
import org.apache.spark.sql.expressions.{Window => W}
import org.apache.spark.sql.{functions => F}
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.GlueArgParser

import com.amazonaws.services.glue.DynamicFrame

import com.amazonaws.regions.Regions
import com.amazonaws.services.glue.model._

import com.amazonaws.services.glue.{AWSGlue, AWSGlueClient}
import scala.collection.JavaConverters.{mapAsJavaMapConverter, seqAsJavaListConverter}
import com.amazonaws.services.sagemaker.sparksdk.IAMRole

val sc = spark.sparkContext
val glueContext: GlueContext = new GlueContext(sc)

val region = Regions.fromName("us-east-1")

// 创建AWS Glue客户端的函数
def glueClient(region: Regions):
AWSGlue = AWSGlueClient.builder().withRegion(region).build()

val glue = glueClient(region = region)

val getConnectionRequest: GetConnectionRequest = new GetConnectionRequest()
  .withName("嵌入式连接的名称")
  .withHidePassword(false)

val getConnectionResult: GetConnectionResult =  glue.getConnection(getConnectionRequest)

val connection: Connection = getConnectionResult.getConnection()

val connectionProperties = connection.getConnectionProperties()

// 现在您可以提取所需的属性(请参阅https://github.com/aws/aws-sdk-java/blob/2d73c9847a327eea2f673a4cd2b3449e433649e5/aws-java-sdk-glue/src/main/java/com/amazonaws/services/glue/model/Connection.java#L687)
// 例如,用户名或密码
val username = connectionProperties.get("USERNAME")
val password = connectionProperties.get("PASSWORD")
英文:

As the error indicates, you need to provide a GetConnectionRequest object to the getConnectionmethod, something like:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._  //import everything from the package    
import org.apache.spark.sql.expressions.{Window =&gt; W} 
import org.apache.spark.sql.{functions =&gt; F}
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.GlueArgParser 

import com.amazonaws.services.glue.DynamicFrame

import com.amazonaws.regions.Regions
import com.amazonaws.services.glue.model._

import com.amazonaws.services.glue.{AWSGlue, AWSGlueClient}
import scala.collection.JavaConverters.{mapAsJavaMapConverter, seqAsJavaListConverter}
import com.amazonaws.services.sagemaker.sparksdk.IAMRole

val sc = spark.sparkContext
val glueContext: GlueContext = new GlueContext(sc)

val region = Regions.fromName(&quot;us-east-1&quot;)

// Function to create AWS glue client
def glueClient(region: Regions):
AWSGlue = AWSGlueClient.builder().withRegion(region).build()

val glue = glueClient(region =region)

val getConnectionRequest: GetConnectionRequest = new GetConnectionRequest()
  .withName(&quot;name-of-embedded-connection&quot;)
  .withHidePassword(false)

val getConnectionResult: GetConnectionResult =  glue.getConnection(getConnectionRequest)

val connection: Connection = getConnectionResult.getConnection()

val connectionProperties = connection.getConnectionProperties()

// Now you can extract as many properties as you need (see https://github.com/aws/aws-sdk-java/blob/2d73c9847a327eea2f673a4cd2b3449e433649e5/aws-java-sdk-glue/src/main/java/com/amazonaws/services/glue/model/Connection.java#L687)
// For instance, the username or password
val username = connectionProperties.get(&quot;USERNAME&quot;)
val password = connectionProperties.get(&quot;PASSWORD&quot;)

huangapple
  • 本文由 发表于 2020年8月14日 02:29:37
  • 转载请务必保留本文链接:https://go.coder-hub.com/63401202.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定