-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support mutable config for all kinds of shadowed services #142
Comments
This was referenced Sep 29, 2022
which contains 3 major fields:
{
"app": {
"metadata": {
"name": "order-mesh",
"namespace": "mesh-service",
"uid": "38071fc0-19a3-4a6d-b2c4-70c8d58c5e1b",
"resourceVersion": "11688463",
"generation": 2,
"creationTimestamp": "2022-07-15T13:43:38Z",
"annotations": {
"deployment.kubernetes.io/revision": "2",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"mesh.megaease.com/service-name\":\"order-mesh\"},\"name\":\"order-mesh\",\"namespace\":\"mesh-service\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"order-mesh\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"order-mesh\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"SERVICE_NAME\",\"value\":\"order-mesh\"},{\"name\":\"ZIPKIN_SERVER_URL\",\"value\":\"https://dev.megaease.cn/zipkin\"}],\"image\":\"megaease/consuldemo:latest\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"order-mesh\",\"ports\":[{\"containerPort\":80}]}]}}}}\n",
"mesh.megaease.com/service-name": "order-mesh"
},
"managedFields": [
{
"manager": "kubectl-client-side-apply",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-07-15T13:43:38Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:kubectl.kubernetes.io/last-applied-configuration": {},
"f:mesh.megaease.com/service-name": {}
}
},
"f:spec": {
"f:progressDeadlineSeconds": {},
"f:replicas": {},
"f:revisionHistoryLimit": {},
"f:selector": {},
"f:strategy": {
"f:rollingUpdate": {
".": {},
"f:maxSurge": {},
"f:maxUnavailable": {}
},
"f:type": {}
},
"f:template": {
"f:metadata": {
"f:labels": {
".": {},
"f:app": {}
}
},
"f:spec": {
"f:containers": {
"k:{\"name\":\"order-mesh\"}": {
".": {},
"f:env": {
".": {},
"k:{\"name\":\"SERVICE_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"ZIPKIN_SERVER_URL\"}": {
".": {},
"f:name": {},
"f:value": {}
}
},
"f:image": {},
"f:imagePullPolicy": {},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":80,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:protocol": {}
}
},
"f:resources": {},
"f:terminationMessagePath": {},
"f:terminationMessagePolicy": {}
}
},
"f:dnsPolicy": {},
"f:restartPolicy": {},
"f:schedulerName": {},
"f:securityContext": {},
"f:terminationGracePeriodSeconds": {}
}
}
}
}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-10-01T07:54:55Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:conditions": {
".": {},
"k:{\"type\":\"Available\"}": {
".": {},
"f:type": {}
},
"k:{\"type\":\"Progressing\"}": {
".": {},
"f:lastTransitionTime": {},
"f:status": {},
"f:type": {}
}
}
}
}
},
{
"manager": "kubectl-edit",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-10-01T08:38:24Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {
"f:template": {
"f:spec": {
"f:volumes": {
"k:{\"name\":\"config\"}": {
".": {},
"f:configMap": {
".": {},
"f:defaultMode": {},
"f:name": {}
},
"f:name": {}
}
}
}
}
}
}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-10-07T03:38:38Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:deployment.kubernetes.io/revision": {}
}
},
"f:status": {
"f:availableReplicas": {},
"f:conditions": {
"k:{\"type\":\"Available\"}": {
"f:lastTransitionTime": {},
"f:lastUpdateTime": {},
"f:message": {},
"f:reason": {},
"f:status": {}
},
"k:{\"type\":\"Progressing\"}": {
"f:lastUpdateTime": {},
"f:message": {},
"f:reason": {}
}
},
"f:observedGeneration": {},
"f:readyReplicas": {},
"f:replicas": {},
"f:updatedReplicas": {}
}
},
"subresource": "status"
}
]
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "order-mesh"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "order-mesh"
}
},
"spec": {
"volumes": [
{
"name": "agent-volume",
"emptyDir": {}
},
{
"name": "sidecar-volume",
"emptyDir": {}
},
{
"name": "config",
"configMap": {
"name": "game-demo",
"defaultMode": 420
}
}
],
"initContainers": [
{
"name": "initializer",
"image": "megaease/easeagent-initializer:latest",
"command": [
"sh",
"-c",
"set -e\ncp -r /easeagent-volume/* /agent-volume\n\necho 'name: order-mesh\ncluster-name: easemesh-control-plane\ncluster-role: secondary\ncluster-request-timeout: 10s\ncluster:\n primary-listen-peer-urls: http://easemesh-control-plane-service.easemesh:2380\nlabels:\n mesh-alive-probe: http://localhost:9900/health\n mesh-application-port: 80\n mesh-service-labels: \n mesh-service-name: order-mesh\n' > /sidecar-volume/sidecar-config.yaml"
],
"resources": {},
"volumeMounts": [
{
"name": "agent-volume",
"mountPath": "/agent-volume"
},
{
"name": "sidecar-volume",
"mountPath": "/sidecar-volume"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"containers": [
{
"name": "order-mesh",
"image": "megaease/consuldemo:latest",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"env": [
{
"name": "SERVICE_NAME",
"value": "order-mesh"
},
{
"name": "ZIPKIN_SERVER_URL",
"value": "https://dev.megaease.cn/zipkin"
},
{
"name": "JAVA_TOOL_OPTIONS",
"value": " -javaagent:/agent-volume/easeagent.jar -Deaseagent.log.conf=/agent-volume/log4j2.xml "
}
],
"resources": {},
"volumeMounts": [
{
"name": "agent-volume",
"mountPath": "/agent-volume"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
},
{
"name": "easemesh-sidecar",
"image": "megaease/easegress:server-sidecar",
"command": [
"/bin/sh",
"-c",
"/opt/easegress/bin/easegress-server -f /sidecar-volume/sidecar-config.yaml"
],
"ports": [
{
"name": "sidecar-ingress",
"containerPort": 13001,
"protocol": "TCP"
},
{
"name": "sidecar-egress",
"containerPort": 13002,
"protocol": "TCP"
},
{
"name": "sidecar-eureka",
"containerPort": 13009,
"protocol": "TCP"
}
],
"env": [
{
"name": "APPLICATION_IP",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "status.podIP"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "sidecar-volume",
"mountPath": "/sidecar-volume"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": "25%",
"maxSurge": "25%"
}
},
"revisionHistoryLimit": 10,
"progressDeadlineSeconds": 600
},
"status": {
"observedGeneration": 2,
"replicas": 1,
"updatedReplicas": 1,
"readyReplicas": 1,
"availableReplicas": 1,
"conditions": [
{
"type": "Progressing",
"status": "True",
"lastUpdateTime": "2022-10-01T08:38:26Z",
"lastTransitionTime": "2022-07-15T13:43:38Z",
"reason": "NewReplicaSetAvailable",
"message": "ReplicaSet \"order-mesh-65bb5976cc\" has successfully progressed."
},
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2022-10-07T03:38:38Z",
"lastTransitionTime": "2022-10-07T03:38:38Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
}
]
}
},
"configMaps": [
{
"metadata": {
"name": "game-demo",
"namespace": "mesh-service",
"uid": "6c2d970e-cfb2-4c3f-bd6b-92a6356771fb",
"resourceVersion": "11260453",
"creationTimestamp": "2022-10-01T08:37:41Z",
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"data\":{\"game.properties\":\"enemy.types=aliens,monsters\\nplayer.maximum-lives=5 \\n\",\"player_initial_lives\":\"3\",\"ui_properties_file_name\":\"user-interface.properties\",\"user-interface.properties\":\"color.good=purple\\ncolor.bad=yellow\\nallow.textmode=true\\n\"},\"kind\":\"ConfigMap\",\"metadata\":{\"annotations\":{},\"name\":\"game-demo\",\"namespace\":\"mesh-service\"}}\n"
},
"managedFields": [
{
"manager": "kubectl-client-side-apply",
"operation": "Update",
"apiVersion": "v1",
"time": "2022-10-01T08:37:41Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:data": {
".": {},
"f:game.properties": {},
"f:player_initial_lives": {},
"f:ui_properties_file_name": {},
"f:user-interface.properties": {}
},
"f:metadata": {
"f:annotations": {
".": {},
"f:kubectl.kubernetes.io/last-applied-configuration": {}
}
}
}
}
]
},
"data": {
"game.properties": "enemy.types=aliens,monsters\nplayer.maximum-lives=5 \n",
"player_initial_lives": "3",
"ui_properties_file_name": "user-interface.properties",
"user-interface.properties": "color.good=purple\ncolor.bad=yellow\nallow.textmode=true\n"
}
}
],
"secrets": null
}
The added major fields are:
ss.json looks like: {
"kind": "ShadowService",
"name": "shadow-001",
"namespace": "mesh-service",
"serviceName": "order-mesh",
"envs": {
"MY_ENV": "FreeMine",
"ZIPKIN_SERVER_URL": "http://fake_addr:12030"
},
"configMaps": [
{
"apiVersion": "v1",
"kind": "ConfigMap",
"metadata": {
"labels": {
"esl-component": "nginx1",
"esl-stackId": "easetest2-default-docker_desktop",
"esl-stackName": "easetest2",
"esl-version": "a4b870a2-2a4e-451b-b143-fb0f9a3bf207"
},
"name": "nginx1",
"namespace": "default"
}
}
],
"secrets": [
{
"apiVersion": "v1",
"data": {
"ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1ERXhPVEE1TkRrd00xb1hEVE15TURFeE56QTVORGt3TTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS1czCkdHTzE1OURtNVI0dnMyaUNsUXIxcExMd1lLK2NKc3RBT0pzbGw0eEZQYk5oTUVNYlRjSHJrazhpYnRkclVGemIKVU45WXhEN3RMTU04dWpLWnRXTldFdDMydExJNUhJTzczYTFtRUpFSnI1UG9EcG9GTHdSYWxPWXFNNG54RjI5awp1M1hJbkh6QUV2MzFyakdhQ0pBeFBuTmdLQlZrV0hNL0c2RlBxRWZpSk9WK3hiMXdKRmdkZE1UUEZrNjEyVmtnCktxR0wxRUpUNFRPSkZzS0pvSHk2M2VFblVQOUM5NitJbmZ3RmFGS3huQWdzMENnYnpMYmp1ZlBVa052a3pTU0sKLzk2elRxMkhXc05DMjlNcFhvOGs3clY0L2xYeGhuSWEzdVo1dGZsQnU1dTExVUMzcGR3VWU3ZTFkOTVYZWU0ZwpENFhkbTlFTGNBb0NhbVcvNURVQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZNWFhjMnFGSVZsQVh5NFlXVFBSOXlLWEIzMjFNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ3dEb1JpQzRRMmFpZTRIK1hHQQpoL2pXOGQ0cmkrL0kzMFhNQ05VNVUvK0Nib290azBTc0JhTjBQOVYxeXBkeDZDOXg3a2FSUGRqSkJVVU1qVGJuClZiZkJKV1VVNFpwcHJHMDRXU1I3NjBYWS91Sk9VVGtKWkdURXlIbG1FMk40eGdTYVpmWUhmZU1YekV5RTdoOWEKTUNYRFE0dFViajZQejl4WTFWZTRXQnJvdldxWFN4WVUwemRGMUNrbG5qa3EwU1lQcnVJRUdDNDdENDcrNks5aQpRSjBhQWFRQ29RY01xdnVDODVUd0VqeHluQnY3NStTYnVkSEkxVk1UQzJCUGFBMWlxRHluNDF4d3hiT2d4SForCnVZdUdtUVB0cnNHYjF4QmRrZ1hXYUlUSmtJOGxHYWpmNTFTN25BWHdpeVp1SE10MS9WTXUzTG00T2tqRU1xWHIKcXNvPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
"namespace": "ZGVmYXVsdA==",
"token": "ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltWmpNall0Y2xoWVgxSndVRlJuVDFoTlUybDFYMnBrZFdWSlFWRnBTRzVYTWtSS01WZGhjblpUVlRRaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbVJsWm1GMWJIUXRkRzlyWlc0dFkyUjBjbk1pTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWkdWbVlYVnNkQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJbUZrWVRFd05qWXpMVEJpWTJFdE5EYzRZeTA0TVdFMUxUazJZVE00TXpsbE1tTmhaQ0lzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwa1pXWmhkV3gwT21SbFptRjFiSFFpZlEuYmFwY093RE5FYlhOaDM1bkdRcnlEU2tFQzh2UmdFenc0OERBNFdXMmxmSWFIN21Zb2RYWGNORS1iek9PMzZ4eldkWndtTm53ODRoYlQ1LWE5NHJtTXB1QlA2MVhucno2VzZNcUFvcmxSMWlGcmxJUFVsb3J3NWhEYS1xUlRDRlMzNDVQN2F5ZjNDM1ZueDRrYVJFYzZUbk5PWGdtNlU0RE1hQXZqbm9NTG5CSWlCVkdzUVhvUWxlREVTLUpVNEp5VUhVeWRKT2kxR244VnV4MjlXcDhtV3BiUHA0UGhOaXhaX2U2dmFjajRWNmF1N0NObm5URE82MVBvZzlDeGdPRXpxbklCZlpJLWs3bUdpQVFNT2RCZG0yWUJJSUUwZjBvTExqWnUwamtBSlk1MFpGMVNVOVZQYVo4NUprVUtLWFNieGo5cWFvUVZrcEd4Q2RsWDJsbUtR"
},
"kind": "Secret",
"metadata": {
"annotations": {
"kubernetes.io/service-account.name": "default",
"kubernetes.io/service-account.uid": "ada10663-0bca-478c-81a5-96a3839e2cad"
},
"name": "default-token-cdtrs",
"namespace": "default"
},
"type": "kubernetes.io/service-account-token"
}
]
}
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
As EaseMesh has started to support non-java applications, the shadow service only supports java for now. So we want to expand our product to include as many features as possible for non-java services, shadow service is the next choice.
Shadow service supports changing the addresses of some middleware, for example, the process needs to change production endpoints to staging/testing ones in case of disturbing the production lines.
We inject all containers with sidecar and EaseAgent[1]. But only applications running in JVM will load and launch EaseAgent. And sidecar will notify EaseAgent of the shadowed middleware information through sidecar protocol[2].
On the other hand, there's no explicit information about whether the internal running application is a Java application or not. So we must distinguish the type of applications before passing corresponding configs.
So we can break down the things that we must do to support mutable config for all kinds of shadowed services:
Proposal
In the need to generalize this feature, we should expand the existing sidecar protocol for the language-insensitive guarantee.
Agent Type
There are mainly two kinds of learning agent type of applications:
This is the simplest solution, but it actually relies on the users' awareness, which could give us the wrong information.
http://localhost:9900/agent-info
, which returns the agent information including agent type, such asThis method would give us the real result and need no awareness from users. But it could bring complexity and make inconsistency among different service instances, which might give inconsistent agent types in some cases (although it's the responsibility of users to prevent it from happening).
So in another perspective, the manual solution is to add static service-level information, but the automatic one is to add dynamic service-instance-level information.
IMHO, the automatic one is better in the case of rigid standards.
Deliver Config
Currently, only EaseAgent can take up config(only the observability part). Besides that, we prepare to support map other configs into the application which means the container in terms of Kubernetes env.
Config categories could be these below:
shadow-xxx-configmap-01
), and then be mutated by users.The more work for ConfigMap and Secret is the extra lifecycle management while deleting Shadow Service. An example would be
shadowed into:
As we can see, the format of copies is
{configmap/secret name}-{deployment/statefulset name}-shadow
, which contains the content which is coped and changed by users (if they want).The reason we add
{deployment/statefulset name}
into the shadowed config name is that the original configmap/secret may be shared by multiple deployments/statefulsets. As we split them totally, the cleaning of one shadow resource won't affect others. As we see, this brings a certain amount of complexity as a cost.The added APIs would be:
Reference
[1] https://github.com/megaease/easeagent
[2] https://github.com/megaease/easemesh/blob/main/docs/sidecar-protocol.md
The text was updated successfully, but these errors were encountered: