runtime error, while cluster is found.

[stafot]

{
  "errorMessage": "runtime error: invalid memory address or nil pointer dereference",
  "errorType": "errorString",
  "stackTrace": [
    {
      "path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go",
      "line": 27,
      "label": "(*Function).Invoke.func1"
    },
    {
      "path": "runtime/asm_amd64.s",
      "line": 573,
      "label": "call32"
    },
    {
      "path": "runtime/panic.go",
      "line": 502,
      "label": "gopanic"
    },
    {
      "path": "runtime/panic.go",
      "line": 63,
      "label": "panicmem"
    },
    {
      "path": "runtime/signal_unix.go",
      "line": 388,
      "label": "sigpanic"
    },
    {
      "path": "lambda-eks-sample/main.go",
      "line": 109,
      "label": "(*ClusterConfig).loadConfig"
    },
    {
      "path": "lambda-eks-sample/main.go",
      "line": 51,
      "label": "handler"
    },
    {
      "path": "runtime/asm_amd64.s",
      "line": 577,
      "label": "call512"
    },
    {
      "path": "reflect/value.go",
      "line": 447,
      "label": "Value.call"
    },
    {
      "path": "reflect/value.go",
      "line": 308,
      "label": "Value.Call"
    },
    {
      "path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go",
      "line": 111,
      "label": "newHandler.func1"
    },
    {
      "path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go",
      "line": 22,
      "label": "lambdaHandler.Invoke"
    },
    {
      "path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go",
      "line": 59,
      "label": "(*Function).Invoke"
    },
    {
      "path": "runtime/asm_amd64.s",
      "line": 574,
      "label": "call64"
    },
    {
      "path": "reflect/value.go",
      "line": 447,
      "label": "Value.call"
    },
    {
      "path": "reflect/value.go",
      "line": 308,
      "label": "Value.Call"
    },
    {
      "path": "net/rpc/server.go",
      "line": 384,
      "label": "(*service).call"
    },
    {
      "path": "runtime/asm_amd64.s",
      "line": 2361,
      "label": "goexit"
    }
  ]
}

START RequestId: 4e8ecdc7-a548-11e8-9f64-250e268ae38f Version: $LATEST
time="2018-08-21T13:44:20Z" level=info msg="Looking up EKS cluster" cluster=xxx
time="2018-08-21T13:44:22Z" level=info msg="Found cluster" cluster=xxx
time="2018-08-21T13:44:22Z" level=debug msg="Cluster details" cluster="<nil>"
runtime error: invalid memory address or nil pointer dereference: errorString
[{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go","line":27,"label":"(*Function).Invoke.func1"},{"path":"runtime/asm_amd64.s","line":573,"label":"call32"},{"path":"runtime/panic.go","line":502,"label":"gopanic"},{"path":"runtime/panic.go","line":63,"label":"panicmem"},{"path":"runtime/signal_unix.go","line":388,"label":"sigpanic"},{"path":"lambda-eks-sample/main.go","line":109,"label":"(*ClusterConfig).loadConfig"},{"path":"lambda-eks-sample/main.go","line":51,"label":"handler"},{"path":"runtime/asm_amd64.s","line":577,"label":"call512"},{"path":"reflect/value.go","line":447,"label":"Value.call"},{"path":"reflect/value.go","line":308,"label":"Value.Call"},{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go","line":111,"label":"newHandler.func1"},{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go","line":22,"label":"lambdaHandler.Invoke"},{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go","line":59,"label":"(*Function).Invoke"},{"path":"runtime/asm_amd64.s","line":574,"label":"call64"},{"path":"reflect/value.go","line":447,"label":"Value.call"},{"path":"reflect/value.go","line":308,"label":"Value.Call"},{"path":"net/rpc/server.go","line":384,"label":"(*service).call"},{"path":"runtime/asm_amd64.s","line":2361,"label":"goexit"}]
END RequestId: 4e8ecdc7-a548-11e8-9f64-250e268ae38f
REPORT RequestId: 4e8ecdc7-a548-11e8-9f64-250e268ae38f	Duration: 1241.58 ms	Billed Duration: 1300 ms 	Memory Size: 128 MB	Max Memory Used: 44 MB	
runtime error: invalid memory address or nil pointer dereference
errorString

Getting the above errors while trying to run a test event.
The privileges of my role are proper I think, because I tested them outside of AWS lambda environment and work as expected.
Any idea if I misconfigured something?
Thanks, in advance.

[stafot]

@chankh time="2018-08-21T13:44:22Z" level=debug msg="Cluster details" cluster="<nil>" It looks like my role has not the ability to describe EKS cluster, while I attached the relevant IAM Policy. Initially also my role had more restricted K8s rbac than system:masters but seems that the problem is in a previous state on describing cluster. Any Ideas why attaching AmazonEKSClusterPolicy does not solve this failure?

[chankh]

@stafot you will need the eks:DescribeCluster permission in your Lambda's execution role. I have just updated the README with this info. Thank you for pointing out.

[stafot]

@chankh AmazonEKSClusterPolicy does not have eks:DescribeCluster so this was the reason of not working. Thanks for your answer and your documentation update.

[stafot]

@chankh In a new lambda function I recently created, get the same error, but not always, randomly. Is self-cured so it is not a big deal, but it is very strange. Based on this I checked my code and your eksauth util and all error controls look as expected. I am not reopening because as I said it happens randomly and not affects the reliability of my function, just sharing it if you have any ideas about the cause.

[chankh]

@stafot the cluster name is retrieved from environment variables, so what I can suggest as a way to debug is to print that CLUSTER_NAME environment variable out in the log, and whenever it fails, check to see if the value is set properly.

[stafot]

@chankh thanks I will try it.

[stafot]

From cloudwatch logs:

18:47:45
time="2019-03-16T18:47:45Z" level=info msg="Found cluster" cluster=my_cluster
18:47:45
time="2019-03-16T18:47:45Z" level=debug msg="Cluster details" cluster="<nil>"

@chankh Seems that c.ClusterName is set but result.cluster doesn't being set. ref. code
I assume needs to be added a line that will log the aws api error to have more visibility on what gone wrong. Do you agree?Are you ok if I raise a PR with this addition?

[chankh]

@stafot you are right, the errors in loadConfig() needs to be propagated back to the calling function. Looking at your logs, does your Lambda function has the eks:DescribeCluster IAM permission?

[stafot]

@chankh Yes it has the proper permissions. Error is shown in random pattern. It is a cron and 9/10 works fine but ~ 1/10 fails.

[stafot]

@chankh I will raise the relevant PR to increase my visibility to the problem.

[stafot]

time="2019-03-26T18:47:45Z" level=error msg="RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe" cluster=my_cluster

18:47:45
time="2019-03-26T18:47:45Z" level=fatal msg="Unable to load Kubernetes Client Config: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe" error="Unable to load Kubernetes Client Config: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe"

18:47:45

2019/03/26 18:47:45 unexpected EOF

18:47:45

2019/03/26 18:47:45 unexpected EOF

The relevant errors after applied logging enhancement to the lambdas of mine.

/cc @chankh

[stafot]

@chankh
I have found similar errors in several places reported. f.e helm

A reasonable explanation of the issue seems to be this