I am using Kubernetes for deployment, but not able to retain data

[221bshashank]

Describe the bug

I am using Kubernetes for deployment but afraid if the node goes down my data will be lost, I want to retain data for last one week, by somehow mounting of volume.
I am using your base image and only specifying the yaml configuration as I am not able to go in the Image/container. I wanted to know does gatus exposes some functionality for this or not..

What do you see?

I am using Kubernetes for deployment but afraid if the node goes down my data will be lost, I want to retain data for last one week, by somehow mounting of volume.

What do you expect to see?

I want to retain the data even if node goes down

List the steps that must be taken to reproduce this issue

Just kill the node and data is gone

Version

latest

Additional information

pls comment ASAP, need this on enterprise level

[antikilahdjs]

The docs have this full explanation about it. You can configure an external database as such as Postgres. To production environment the basis is to use a CSI with HA in those disks. If your cluster goes down the storage still intact

[221bshashank]

@antikilahdjs I can see some examples for using storage with docker compose configuration, could you pls point out if there's any documentation for using storage with Kubernetes.

[Chittiinfo]

---

We hosted gatus app on our EKS and we are able to monitor all our live URLs without any issues. But Health check API data is getting deleted when gatus pod get crashed\delete and due to that we are unable to refer old data if thr r any issues with specific API URL in the past from gatus console. I have tried with below deployment yaml which have both gatus and postgres DB containers with ebs based PVC cliam but still data is not retaining when gatus pod goes down. Anyone tried to deploy gatus tool with PVC to retain health chks api data?

kind: ConfigMap
metadata:
  name: gatus
  namespace: kube-system
data:
  storage: |
    type: postgres
    path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"
  config.yaml: |
    metrics: true
    endpoints:
      - name: website
        url: https://twin.sh/health
        interval: 5m
        conditions:
          - "[STATUS] == 200"
          - "[BODY].status == UP"

      - name: github
        url: https://api.github.com/healthz
        interval: 5m
        conditions:
          - "[STATUS] == 200"

      - name: cat-fact
        url: "https://cat-fact.herokuapp.com/facts/random"
        interval: 5m
        conditions:
          - "[STATUS] == 200"
          - "[BODY].deleted == false"
          - "len([BODY].text) > 0"
          - "[BODY].text == pat(*cat*)"
          - "[STATUS] == pat(2*)"
          - "[CONNECTED] == true"

      - name: example
        url: https://example.com/
        conditions:
          - "[STATUS] == 200"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gatus
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gatus
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gatus
  template:
    metadata:
      name: gatus
      namespace: kube-system
      labels:
        app: gatus
    spec:
      serviceAccountName: gatus
      terminationGracePeriodSeconds: 5
      containers:
      - env:
        - name: POSTGRES_DB
          value: gatus
        - name: POSTGRES_USER
          value: username
        - name: POSTGRES_PASSWORD
          value: password
        image: postgres
        imagePullPolicy: Always
        name: postgres
        ports:
        - containerPort: 5432
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/postgresql
          name: gatus-db
      - image: twinproduction/gatus
        imagePullPolicy: IfNotPresent
        name: gatus
        ports:
            - containerPort: 8080
              name: http
              protocol: TCP
        resources:
            limits:
              cpu: 250m
              memory: 100M
            requests:
              cpu: 50m
              memory: 30M
        readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
        livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 5
        volumeMounts:
            - mountPath: /config
              name: gatus-config
      volumes:
        - configMap:
            name: gatus
          name: gatus-config
        - name: gatus-db 
          persistentVolumeClaim:
           claimName: gatus  ### created an EBS based PVC
---
apiVersion: v1
kind: Service
metadata:
  name: gatus
  namespace: kube-system
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: gatus```
 

[antikilahdjs]

Hello,

I don't if I got it but it is about gatus or pvc? For example, in gatus doc it is possible to point it to postgres and I did it before then it worked fine but I've decided put it in my CSI because I have the PROD environment with WEKA. My cluster has large storage replication with the same point of access.

If you use the OpenEBS for example it will mount in each node and in that case if the server crash you will lose it .

Today I using this configuration below.

  storage:
    caching: false
    path: /data/gatus.db
    type: sqlite

Inside of the pod the path is /data but for in my CSI is /data/weka and all my gatus has a PVC attached then my data will be instacted, it is the normal way for all deployment and so on

If you need put it in own database the gatus just support postgres

storage:
  type: postgres
  path: "postgres://user:password@127.0.0.1:5432/gatus?sslmode=disable"

[TwiN]

Can you format your configuration properly? (Wrap it using ```)

At a glance, it seems like the problem has to do with your storage configration.

It should be

storage:
  type: postgres
  path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"

But instead, you're putting it in a storage.yaml file:

storage.yaml: |
  type: postgres
  path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"

[Chittiinfo]

Can you format your configuration properly? (Wrap it using ```)

At a glance, it seems like the problem has to do with your storage configration.

It should be

storage:
  type: postgres
  path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"

But instead, you're putting it in a storage.yaml file:

storage.yaml: |
  type: postgres
  path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"

I have tried with below config as u suggested but still data is not retaining. Any other suggestions? i have updated the complete using code in my previous post.

  storage: |
    type: postgres
    path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"```

[Chittiinfo]

Hello,

I don't if I got it but it is about gatus or pvc? For example, in gatus doc it is possible to point it to postgres and I did it before then it worked fine but I've decided put it in my CSI because I have the PROD environment with WEKA. My cluster has large storage replication with the same point of access.

If you use the OpenEBS for example it will mount in each node and in that case if the server crash you will lose it .

Today I using this configuration below.

  storage:
    caching: false
    path: /data/gatus.db
    type: sqlite

Inside of the pod the path is /data but for in my CSI is /data/weka and all my gatus has a PVC attached then my data will be instacted, it is the normal way for all deployment and so on

If you need put it in own database the gatus just support postgres

storage:
  type: postgres
  path: "postgres://user:password@127.0.0.1:5432/gatus?sslmode=disable"

Currently gatus dashboard is not showing past data if gatus pod goes down and i am trying to ensure my gatus dashboard shows old data even gatus pod get terminated/re-created. I am trying to retain the data with postgres db container which "/var/lib/postgresql" folder is mounted on ebs based PVC. I can see DB level is retaining on pv cliam but gatus container is not storing any data on postgres data folder. I believe gatus is not able to use postgres container for some reason to store the data.
I am using below deploy and config yamls in my case, not sure wht i m missing here. Would you be able to share ur sample deployment and configmap yamls for referennce?

kind: Deployment
metadata: 
  name: gatus
  namespace: gatus

spec:
  replicas: 1
  selector:
    matchLabels:
      app: gatus
  template:
    metadata:
      labels:
        app: gatus
      name: gatus
      namespace: gatus
    spec:
      containers:
      - env:
        - name: POSTGRES_DB
          value: gatus
        - name: POSTGRES_USER
          value: username
        - name: POSTGRES_PASSWORD
          value: password
        image: postgres
        name: postgres
        ports:
        - containerPort: 5432
          protocol: TCP
        volumeMounts:
        - mountPath: /var/lib/postgresql
          name: gatus-db
      - env:
        - name: USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: gatus-cred
        - name: PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: gatus-cred
        image: twinproduction/gatus
        name: gatus
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        volumeMounts:
        - mountPath: /config
          name: gatus-config
      volumes:
      - configMap:
          defaultMode: 420
          name: gatus
        name: gatus-config
      - name: gatus-db
        persistentVolumeClaim:
          claimName: gatus```

---------

```kind: ConfigMap
metadata:
  name: gatus
  namespace: kube-system
data:
  storage: |
    type: postgres
    path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"
  config.yaml: |
    metrics: true
    endpoints:
      - name: website
        url: https://twin.sh/health
        interval: 5m
        conditions:
          - "[STATUS] == 200"
          - "[BODY].status == UP"

      - name: github
        url: https://api.github.com/healthz
        interval: 5m
        conditions:
          - "[STATUS] == 200"```
------

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
  name: gatus
  namespace: gatus
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: gp2
  volumeMode: Filesystem
  volumeName: pvc-437dabfd-69df-4a0b-89cb-4db632992247
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 8Gi
  phase: Bound

[TwiN]

It's because you're not formatting the configuration properly.

It should look like this:

kind: ConfigMap
metadata:
  name: gatus
  namespace: kube-system
data:
  config.yaml: |
    metrics: true

    storage:
      type: postgres
      path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"

    endpoints:
      - name: website
        url: https://twin.sh/health
        interval: 5m
        conditions:
          - "[STATUS] == 200"
          - "[BODY].status == UP"

      - name: github
        url: https://api.github.com/healthz
        interval: 5m
        conditions:
          - "[STATUS] == 200"

[Chittiinfo]

I have tried with above suggested configmap code but gatus container is failing with "CrashLoopBackoff" error now. Gatus container is trying to lookup 172.20.0.10 for postgres but its unable to resolve it. Not sure why its looking for 172.x.x.x series ip when my postgres is running on 10.x.x.x ip series. Any idea on this error?
FYI, without storage configuration in above CM my gatus app is working fine. All i am trying to retain the gatus data and thats where i am seeing all these issues.

[TwiN]

To clarify, you didn't leave

path: "postgres://username:password@postgres:5432/gatus?sslmode=disable"

exactly as-is, right?

You have to substitute username:password@postgres:5432/gatus with your explicit values, so if postgres was running on 123.123.123.123:5432, my username was hello, password was world and database name was mydb, it would be

path: "postgres://hello:world@123.123.123.123:5432/mydb?sslmode=disable"