Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

audit_info lookupUser with empty content leads to parse error #164

Open
estein9825 opened this issue Feb 23, 2023 · 7 comments
Open

audit_info lookupUser with empty content leads to parse error #164

estein9825 opened this issue Feb 23, 2023 · 7 comments
Labels
bug Something isn't working

Comments

@estein9825
Copy link

estein9825 commented Feb 23, 2023 •
edited

Expected vs actual behavior

Somehow the auditing of a user lookup lead to blank info, which caused the following panic exception:

{"level":"info","ts":"2023-02-23T19:49:17.808Z","caller":"server/audit_info.go:31","msg":"lookupUser","attrs":{"AccountID":"","PartnerID":"","OrganizationID":"","Username":"","IsSSO":false,"EnforceSession":false,"SessionType":"","SystemUser":false,"RelayNetwork":false}}
panic: uuid: Parse(): invalid UUID length: 0

goroutine 203 [running]:
github.com/google/uuid.MustParse({0x0, 0x0})
	/go/pkg/mod/github.com/google/uuid@v1.3.0/uuid.go:163 +0xb9
github.com/paralus/paralus/pkg/service.(*accountPermissionService).GetAccount(0xc000313f70, {0x237ad80, 0xc0001e2fc0}, {0x0, 0x0})
	/build/pkg/service/account_permission.go:91 +0x33
github.com/paralus/paralus/server.(*auditInfoServer).LookupUser(0xc000252300, {0x237ad80, 0xc0001e2fc0}, 0x0)
	/build/server/audit_info.go:47 +0x367
github.com/paralus/paralus/proto/rpc/sentry._AuditInformationService_LookupUser_Handler({0x1dbcc60, 0xc000252300}, {0x237ad80, 0xc0001e2fc0}, 0xc0004c5f80, 0x0)
	/build/proto/rpc/sentry/audit_info_grpc.pb.go:91 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006321c0, {0x23b0dd0, 0xc000bada00}, 0xc0002ba480, 0xc000451950, 0x361fe60, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282 +0xccf
google.golang.org/grpc.(*Server).handleStream(0xc0006321c0, {0x23b0dd0, 0xc000bada00}, 0xc0002ba480, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:919 +0x294

Steps to reproduce the bug

  1. Enable auditing and install fluent-bit component
  2. Increase replicas to 3 for both paralus and the relay servers
  3. Bootstrap 3 clusters into the instance
  4. Download kubeconfig and attempt to connect to one of the clusters

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

Dropping the relay server down to 1, turning fluent-bit off, and restoring the database before bringing up paralus again seemed to have gotten it to work.

  • [X ] I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
  • [ X] I'm using the latest version of the project.
@estein9825 estein9825 added bug Something isn't working needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 23, 2023
@estein9825 estein9825 changed the title audit_info lookupUser with null leads to parse error audit_info lookupUser with empty content leads to parse error Feb 23, 2023
@akshay196
Copy link
Member

@estein9825 If you've the failed environment handy, can you share log of relay-tail container part of relay-server pods? There should have an entry inside /audit-logs/audit.log with empty user details. Definitely Paralus must handle empty input for LookupUser rpc. But it is worth checking why a relay audit log is created with empty data.

@estein9825
Copy link
Author

Unfortunately that particular environment was restored and we have not tried to increase our relay server components since then. It might be that it is no longer an issue with the relay patch that I implemented for paralus/relay#43. But I'm not sure. I will keep an eye out and try to grab that info if it happens again.

@divyanjali1
Copy link

@akshay196

We are not able to see audit-logs in paralus dashbaord.

Logs from relay-tail
{"level":"info","ts":"2023-05-08T07:53:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:53:45.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:54:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:54:45.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:55:15.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:55:45.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:56:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:56:45.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:57:15.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:57:45.534Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}
{"level":"info","ts":"2023-05-08T07:58:15.533Z","caller":"tail/tail.go:39","msg":"audit log not updated for 7 days","path":"/audit-logs/relay-server-57c8bfdd8d-79jzw/audit.log"}

logs from /audit-logs/audit.log

{"level":"info","ts":"2023-05-08T08:38:17.389Z","msg":"audit","xid":"chcb9uf9lqnjlm35tipg","method":"GET","url":"/apis/install.istio.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":472,"duration":0.001689512}
{"level":"info","ts":"2023-05-08T08:38:17.393Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiq0","method":"GET","url":"/apis/kafka.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":859,"duration":0.001654422}
{"level":"info","ts":"2023-05-08T08:38:17.397Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiqg","method":"GET","url":"/apis/kinesis.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":466,"duration":0.00170434}
{"level":"info","ts":"2023-05-08T08:38:17.401Z","msg":"audit","xid":"chcb9uf9lqnjlm35tir0","method":"GET","url":"/apis/kms.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":799,"duration":0.001937126}
{"level":"info","ts":"2023-05-08T08:38:17.406Z","msg":"audit","xid":"chcb9uf9lqnjlm35tirg","method":"GET","url":"/apis/lambda.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":474,"duration":0.001971445}
{"level":"info","ts":"2023-05-08T08:38:17.410Z","msg":"audit","xid":"chcb9uf9lqnjlm35tis0","method":"GET","url":"/apis/lambda.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":475,"duration":0.001740026}
{"level":"info","ts":"2023-05-08T08:38:17.414Z","msg":"audit","xid":"chcb9uf9lqnjlm35tisg","method":"GET","url":"/apis/mq.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":806,"duration":0.001833806}
{"level":"info","ts":"2023-05-08T08:38:17.418Z","msg":"audit","xid":"chcb9uf9lqnjlm35tit0","method":"GET","url":"/apis/neptune.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":481,"duration":0.002038169}
{"level":"info","ts":"2023-05-08T08:38:17.422Z","msg":"audit","xid":"chcb9uf9lqnjlm35titg","method":"GET","url":"/apis/notification.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":881,"duration":0.001795342}
{"level":"info","ts":"2023-05-08T08:38:17.427Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiu0","method":"GET","url":"/apis/ram.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":497,"duration":0.001835631}
{"level":"info","ts":"2023-05-08T08:38:17.431Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiug","method":"GET","url":"/apis/rds.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":2087,"duration":0.001690903}
{"level":"info","ts":"2023-05-08T08:38:17.434Z","msg":"audit","xid":"chcb9uf9lqnjlm35tiv0","method":"GET","url":"/apis/route53.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":896,"duration":0.001587222}
{"level":"info","ts":"2023-05-08T08:38:17.438Z","msg":"audit","xid":"chcb9uf9lqnjlm35tivg","method":"GET","url":"/apis/route53resolver.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":1349,"duration":0.001602461}
{"level":"info","ts":"2023-05-08T08:38:17.442Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj00","method":"GET","url":"/apis/secrets.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":357,"duration":0.001508539}
{"level":"info","ts":"2023-05-08T08:38:17.446Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj0g","method":"GET","url":"/apis/secretsmanager.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":472,"duration":0.002077822}
{"level":"info","ts":"2023-05-08T08:38:17.451Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj10","method":"GET","url":"/apis/secretsmanager.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":473,"duration":0.001899756}
{"level":"info","ts":"2023-05-08T08:38:17.455Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj1g","method":"GET","url":"/apis/sfn.aws.crossplane.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":859,"duration":0.00179306}
{"level":"info","ts":"2023-05-08T08:38:17.460Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj20","method":"GET","url":"/apis/telemetry.istio.io/v1alpha1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":504,"duration":0.001896323}
{"level":"info","ts":"2023-05-08T08:38:17.470Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj2g","method":"GET","url":"/apis/aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":781,"duration":0.00204674}
{"level":"info","ts":"2023-05-08T08:38:17.476Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj30","method":"GET","url":"/apis/database.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":881,"duration":0.003256867}
{"level":"info","ts":"2023-05-08T08:38:17.480Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj3g","method":"GET","url":"/apis/aws.crossplane.io/v1alpha3","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":353,"duration":0.001828642}
{"level":"info","ts":"2023-05-08T08:38:17.484Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj40","method":"GET","url":"/apis/networking.istio.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":3304,"duration":0.002164291}
{"level":"info","ts":"2023-05-08T08:38:17.488Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj4g","method":"GET","url":"/apis/iam.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":4058,"duration":0.001892934}
{"level":"info","ts":"2023-05-08T08:38:17.492Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj50","method":"GET","url":"/apis/networking.istio.io/v1alpha3","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":3305,"duration":0.001729394}
{"level":"info","ts":"2023-05-08T08:38:17.497Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj5g","method":"GET","url":"/apis/security.istio.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":1409,"duration":0.002192479}
{"level":"info","ts":"2023-05-08T08:38:17.501Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj60","method":"GET","url":"/apis/s3.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":460,"duration":0.001805061}
{"level":"info","ts":"2023-05-08T08:38:17.505Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj6g","method":"GET","url":"/apis/sns.aws.crossplane.io/v1beta1","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":841,"duration":0.001787196}
{"level":"info","ts":"2023-05-08T08:38:17.509Z","msg":"audit","xid":"chcb9uf9lqnjlm35tj70","method":"GET","url":"/apis/s3.aws.crossplane.io/v1alpha3","query":"timeout=32s","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":493,"duration":0.001902843}
{"level":"info","ts":"2023-05-08T08:38:25.656Z","msg":"audit","xid":"chcba0f9lqnjlm35tj7g","method":"POST","url":"/api/v1/namespaces","query":"fieldManager=kubectl-create&fieldValidation=Strict","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":201,"written":531,"duration":0.015980887}
{"level":"info","ts":"2023-05-08T08:38:28.877Z","msg":"audit","xid":"chcba179lqnjlm35tj80","method":"GET","url":"/api/v1/namespaces","query":"limit=500","serverName":"13de4669-a1f3-4e05-bf63-66a4629afdf6.user.paralus.slvr-corp-corpdemo1.awsdns.internal.das","user":"a=a0d2e395-bb80-4960-88b0-1dc1477593f3/o=0a2699fc-2bec-4b4c-90e2-5bf6c5a28aca/p=1f7bf0a2-ff4a-46f2-a282-1d4bd84bc81a/u=admin-64paralus-46local/is=false/es=false/st=ws/su=false/rn=false/","remoteAddr":"10.188.212.37","statusCode":200,"written":10281,"duration":0.007806086}

@akshay196
Copy link
Member

@divyanjali1 What error or message your are getting on dashboard or from get audit API?
The log message you posted is informative logs which only indicate that system is cleaning up stale audit directories.

If the error message is not related to parse error (as per title of this issue), then it would be better if you open separate issue for it.

@divyanjali1
Copy link

@akshay196 I don't see any logs in dashboard at all and the only logs I see regarding audit-logs are one I have posted above

Can you help me with the specific location for checking the logs ?

@niravparikh05
Copy link
Contributor

@divyanjali1 your issue seems to be different from the current one, I recommend to create a new issue and track the conversations there. It may be confusing to have conversations around two different issues here.

@akshay196 akshay196 removed the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Mar 12, 2024
@akshay196
Copy link
Member

Increase replicas to 3 for both paralus and the relay servers

We are working on #264 for increasing number of relay servers in Paralus. We will address this bug as a part of this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@akshay196 @divyanjali1 @estein9825 @niravparikh05