Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Gemini Experimental #234

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

Support Gemini Experimental #234

wants to merge 9 commits into from

Conversation

DonggeLiu
Copy link
Collaborator

@DonggeLiu DonggeLiu commented Apr 29, 2024 •
edited

This PR supports gemini-experimental, which appears to work well in some cases.
Comparison experiment plan:

  1. Compare gemini-experimental against our default code-bison-32k with the same temperature 0.4.
  2. Run gemini-experimental with its default temperature 1.

@DonggeLiu
Copy link
Collaborator Author

@happy-qop, in case you are curious about the result : )

@DonggeLiu DonggeLiu marked this pull request as draft April 29, 2024 06:15
@DonggeLiu
Copy link
Collaborator Author

DonggeLiu commented Apr 29, 2024 •
edited

gemini-experimental, temperature 0.4:
JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-234-dg
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-04-29-234-dg-comparison/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2024-04-29-234-dg-comparison
BUCKET GS: gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2024-04-29-234-dg-comparison

@DonggeLiu
Copy link
Collaborator Author

gemini-experimental, temperature 1:
JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-234-dg-exp-temp-1
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-04-29-234-dg-exp-temp-1-comparison/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2024-04-29-234-dg-exp-temp-1-comparison
BUCKET GS: gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2024-04-29-234-dg-exp-temp-1-comparison

@DonggeLiu
Copy link
Collaborator Author

code-bison-32k, temperature 1:
JOB: https://console.cloud.google.com/kubernetes/job/us-central1-c/llm-experiment/default/ofg-pr-234-dg-bison-temp-1
REPORT: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-04-29-234-dg-bison-temp-1-comparison/index.html
BUCKET: https://console.cloud.google.com/storage/browser/oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2024-04-29-234-dg-bison-temp-1-comparison
BUCKET GS: gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/ofg-pr/2024-04-29-234-dg-bison-temp-1-comparison

oliverchang
oliverchang reviewed Apr 29, 2024
View reviewed changes
llm_toolkit/models.py Outdated
@@ -350,6 +350,16 @@ class GeminiPro(GeminiModel):
_vertex_ai_model = 'gemini-1.0-pro'


class GeminiExperimental(VertexAIModel):
"""Gemini Experimental, results maybe genius or delightfully unpredictable."""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove the "maybe genius or delightfully unpredictable." part :) I know this is what is shown on Vertex AI, but no need to include it here in our codebase.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

llm_toolkit/models.py Outdated
"""Gemini Experimental, results maybe genius or delightfully unpredictable."""

_max_output_tokens = 8192
context_window = 32760
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal for our current workloads, but is this actually the same 1M context windows as Gemini 1.5?

Copy link
Collaborator Author

@DonggeLiu DonggeLiu Apr 30, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a TODO as the current doc is not very clear.

oliverchang
oliverchang reviewed Apr 30, 2024
View reviewed changes
llm_toolkit/models.py Outdated
@@ -350,6 +350,16 @@ class GeminiPro(GeminiModel):
_vertex_ai_model = 'gemini-1.0-pro'


class GeminiExperimental(VertexAIModel):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to inherit from GeminiModel instead.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks!

@DonggeLiu
Copy link
Collaborator Author

The current quota for gemini-experimental is too low, will try again when the new quota application is approved.

@DonggeLiu
Copy link
Collaborator Author

/gcbrun request_pr_exp.py -n test-gemini-exp -m vertex_ai_gemini-experimental

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oliverchang oliverchang oliverchang left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@DonggeLiu @oliverchang