Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle VertexAI error response 'Text too long' #103

Open
DonggeLiu opened this issue Feb 13, 2024 · 1 comment
Open

Handle VertexAI error response 'Text too long' #103

DonggeLiu opened this issue Feb 13, 2024 · 1 comment

Comments

@DonggeLiu
Copy link
Collaborator

The error is caused by 'Text too long'. Here is the message:
https://pantheon.corp.google.com/logs/query;aroundTime=2024-02-13T12:58:00.000Z;cursorTimestamp=2024-02-13T13:00:23.372992698Z;duration=PT15M;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22oss-fuzz%22%0Aresource.labels.location%3D%22us-central1%22%0Aresource.labels.cluster_name%3D%22llm-experiment%22%0Aresource.labels.namespace_name%3D%22default%22%0Alabels.k8s-pod%2Fbatch_kubernetes_io%2Fcontroller-uid%3D%228f6b14ca-f7a2-4653-85f1-90e542b95e79%22%20severity%3E%3DWARNING%0Atimestamp%3D%222024-02-13T13:00:23.372992698Z%22%0AinsertId%3D%225lnkmvgn576p21g1%22?project=oss-fuzz

The related logs before/after the message:
https://pantheon.corp.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22oss-fuzz%22%0Aresource.labels.location%3D%22us-central1%22%0Aresource.labels.cluster_name%3D%22llm-experiment%22%0Aresource.labels.namespace_name%3D%22default%22%0Alabels.k8s-pod%2Fbatch_kubernetes_io%2Fcontroller-uid%3D%228f6b14ca-f7a2-4653-85f1-90e542b95e79%22%20severity%3E%3DWARNING;cursorTimestamp=2024-02-13T13:00:23.372992698Z;aroundTime=2024-02-13T12:58:00.000Z;duration=PT15M?e=-13802955&mods=logs_tg_prod&project=oss-fuzz

Handling this requires two tasks:

  1. Investigate its root cause. This appears to happen in the code fixing step. We need to understand what the prompt was, and why it is overlong (e.g., Did the error parser parse too much text?)
  2. Capture this error and log it so that the experiment won't break because of it.
@DonggeLiu DonggeLiu changed the title Handle VertexAI error response Handle VertexAI error response 'Text too long' Feb 13, 2024
@cjx10
Copy link
Collaborator

cjx10 commented Mar 8, 2024

Example error:
https://pantheon.corp.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22oss-fuzz%22%0Alabels.%22k8s-pod%2Fbatch_kubernetes_io%2Fjob-name%22%3D%22ofg-pr-148-jim%22;cursorTimestamp=2024-03-08T04:04:35.293228056Z;aroundTime=2024-03-08T04:04:34.772608384Z;duration=PT15S?project=oss-fuzz

Example prompt:
LLM: vertex_ai_code-bison-32k
output-eigen-_zn5eigen6matrixidlin1elin1eli0elin1elin1eed2ev

gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/CodeFixing/text_too_long_prompt.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants