Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support claude ai model #969

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat: support claude ai model #969

wants to merge 5 commits into from

Conversation

cr7258
Copy link
Contributor

@cr7258 cr7258 commented May 18, 2024 •
edited

Ⅰ. Describe what this PR did

Support calude ai model. Claude API documentation: https://docs.anthropic.com/en/api/messages

Ⅱ. Does this pull request fix one issue?

fixes #946

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

docker-compose.yaml

version: '3.7'
services:
  envoy:
    image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/envoy:1.20
    entrypoint: /usr/local/bin/envoy
    # 开启了 debug 级别日志方便调试
    command: -c /etc/envoy/envoy.yaml --component-log-level wasm:debug
    networks:
      - higress-net
    ports:
      - "10000:10000"
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
      - ./plugin.wasm:/etc/envoy/plugin.wasm
networks:
  higress-net: {}

envoy.yaml

admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                scheme_header_transformation:
                  scheme_to_overwrite: https
                stat_prefix: ingress_http
                # Output envoy logs to stdout
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                # Modify as required
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: [ "*" ]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: claude
                            timeout: 300s
                http_filters:
                  - name: claude
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          name: claude
                          vm_config:
                            runtime: envoy.wasm.runtime.v8
                            code:
                              local:
                                filename: /etc/envoy/plugin.wasm
                          configuration:
                            "@type": "type.googleapis.com/google.protobuf.StringValue"
                            value: |
                              {
                                "provider": {
                                  "type": "claude",
                                  # "protocol": "original"  # 设置时使用 Claude AI 协议
                                  "apiTokens": [
                                    "CLAUDE_AI_TOKEN"
                                  ],
                                  "modelMapping": {
                                      "gpt-3": "claude-3-opus-20240229",
                                      "*": "claude-3-sonnet-20240229"
                                   }              
                                }
                              }
                  - name: envoy.filters.http.router
  clusters:
    - name: claude
      connect_timeout: 30s
      type: LOGICAL_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: claude
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: api.anthropic.com
                      port_value: 443
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          "sni": "api.anthropic.com"

使用 OpenAI 协议

非流式请求

curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json"  -d '{
  "model": "claude-3-opus-20240229",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "你好，你是谁？"
    }
  ]
}'

# 响应
{
  "id": "msg_01QiMEE2TN7oKsfJT89DSyUL",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "你好!我是一个人工智能助手,由 Anthropic 公司训练和开发。我是一个对话式 AI 系统,旨在与人友好交流、提供帮助和解答问题。很高兴能和你交谈,不过我只是一个 AI 助手,无法发展出真正的人际关系或情感纽带。但我会尽我所能提供周到、有用和有意义的回应。欢迎你多多提问和探讨,我会诚恳回答。"
      },
      "finish_reason": "stop"
    }
  ],
  "created": 1717387040,
  "model": "claude-3-sonnet-20240229",
  "object": "chat.completion",
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 155,
    "total_tokens": 171
  }
}

流式请求

curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json"  -d '{
  "model": "claude-3-opus-20240229",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "你好，你是谁？"
    }
  ],
  "stream": true
}'

# 响应
{"id":"msg_01Ma9aCKZZtgPDepaRC9rzyd","choices":[{"index":0,"delta":{"role":"assistant"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"你"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"好"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":","}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"我"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"是"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"一"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"个"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"人"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"工"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"智"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"能"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"助"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"理"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":","}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"名"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"字"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"叫"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"Claude"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"。"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"很"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"高"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"兴"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"与"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"你"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"交"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"谈"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"。"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"我"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"是"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"由"}}],"created":1717387087,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"一"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"家"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"名"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"为"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"Anthrop"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"ic"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"的"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"公"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"司"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"开"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"发"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"的"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"一"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"种"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"新"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"型"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"人"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"工"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"智"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"能"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"模"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"型"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"。"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"我"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"的"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"目"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"标"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"是"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"尽"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"最"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"大"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"努"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"力"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"为"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"每"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"个"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"与"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"我"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"交"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"谈"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"的"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"人"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"提"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"供"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"有"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"帮"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"助"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"和"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"友"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"好"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"的"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"服"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"务"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"。"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"请"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"让"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"我"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"知"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"道"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"有"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"任"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"何"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"我"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"可"}}],"created":1717387088,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"以"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"帮"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"助"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"你"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"的"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"地"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"方"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{"content":"。"}}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}
{"choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"created":1717387089,"model":"claude-3-sonnet-20240229","object":"chat.completion.chunk","usage":{}}

stop
stop 接受一个 string 数组，当生成的内容遇到该词时，停止生成。（在 claude 中是 stop_sequence）

curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json"  -d '{
  "model": "claude-3-opus-20240229",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "你好，你是谁？"
    }
  ],
  "stop": ["智能"]
}

# 响应
{
  "id": "msg_01DYKkyG3vatHfZ7bvNwmEqc",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "您好,我是一个由Anthropic公司开发的人工"
      },
      "finish_reason": "stop"
    }
  ],
  "created": 1717387872,
  "model": "claude-3-opus-20240229",
  "object": "chat.completion",
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 21,
    "total_tokens": 37
  }
}

使用 Claude 协议

Claude Streaming 文档：https://docs.anthropic.com/en/api/messages-streaming#streaming-with-sdks

非流式请求

curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json"  -d '{
  "model": "claude-3-opus-20240229",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "你好，你是谁？"
    }
  ]
}'

# 响应
{
  "id": "msg_01QRyVyUFi1QovrwpQmznvHJ",
  "type": "message",
  "role": "assistant",
  "model": "claude-3-opus-20240229",
  "content": [
    {
      "type": "text",
      "text": "你好,我是一个聊天机器人助手,由人工智能公司Anthropic开发和训练。我的名字是Claude,很高兴认识你!我可以就各种话题与你交流,回答问题,给建议等。我会尽最大努力给你有帮助的回复。但我只是一个人工智能,不是真人,希望你理解我的局限性。很高兴能帮到你,欢迎随时来找我聊天!"
    }
  ],
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 16,
    "output_tokens": 142
  }
}

流式请求

curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json"  -d '{
  "model": "claude-3-opus-20240229",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "你好，你是谁？"
    }
  ],
  "stream": true
}'

# 响应
# Claude 返回的原始响应中 } 离前面的字符串就是有空格，这里保持原样没有处理
event: message_start
data: {"type":"message_start","message":{"id":"msg_01XnWGnK1ZpE6WWpG56sET9n","type":"message","role":"assistant","model":"claude-3-opus-20240229","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":16,"output_tokens":1}}   }

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}        }

event: ping
data: {"type": "ping"}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"是"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"一"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"个"}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"由"}         }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"人"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"工"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"智"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"能"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"公"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"司"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Anthrop"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ic"}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"开"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"发"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"的"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"聊"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"天"}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"机"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"器"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"人"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"助"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"手"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"。"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"的"}   }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"名"}              }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"字"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"是"}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Claude"}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"很"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"高"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"兴"}              }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"认"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"识"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"你"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"!"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"}         }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"可"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"以"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"帮"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"助"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"回"}   }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"答"}             }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"问"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"题"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"提"}              }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"供"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"建"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"议"}         }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"和"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"帮"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"助"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"讨"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"论"}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"各"}             }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"种"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"话"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"题"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"。"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"虽"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"然"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"有"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"类"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"似"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"人"}         }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"类"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"的"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"对"}             }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"话"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"能"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"力"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"但"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"请"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"记"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"住"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"}   }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"只"}         }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"是"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"一"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"个"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"AI"}         }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}   }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"不"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"能"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"完"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"全"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"取"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"代"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"人"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"类"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"的"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"交"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"流"}              }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"。"}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"会"}            }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"尽"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"所"}   }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"能"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"提"}          }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"供"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"帮"}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"助"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"希"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"望"}  }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"们"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"相"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"处"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"愉"}     }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"快"}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"!"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0 }

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":134}     }

event: message_stop
data: {"type":"message_stop"  }

stop_sequence

curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json"  -d '{
  "model": "claude-3-opus-20240229",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "你好，你是谁？"
    }
  ],
  "stop_sequences": ["智能"]
}' 

# 响应
{
  "id": "msg_01S4gu2HTDS2AQju9Wbpz8Ch",
  "type": "message",
  "role": "assistant",
  "model": "claude-3-opus-20240229",
  "content": [
    {
      "type": "text",
      "text": "你好!我是Claude,一个由Anthropic公司开发的人工"
    }
  ],
  "stop_reason": "stop_sequence",
  "stop_sequence": "智能",
  "usage": {
    "input_tokens": 16,
    "output_tokens": 23
  }
}

Ⅴ. Special notes for reviews

@cr7258 cr7258 mentioned this pull request May 20, 2024
Merged
@CH3CHO
Copy link
Collaborator

CH3CHO commented May 22, 2024

此外，请解决冲突并同步更新 README.md。

@CH3CHO
Copy link
Collaborator

CH3CHO commented May 22, 2024

Anthropic Claude AI 的 API 模型和 OpenAI 的好像不太一样。请确认能否适配。

@cr7258
Copy link
Contributor Author

cr7258 commented May 22, 2024

@CH3CHO
我的理解是有两个地方我还需要调整：

  1. Claude 和 OpenAI 请求体中的参数不一样，有些 Openai 有的请求体参数，在 Claude 里没有对应的参数，这种情况要怎么处理呢？比如说 OpenAI 有 n 参数，但是 Claude 里好像就没有。
    https://docs.anthropic.com/en/api/messages
    https://platform.openai.com/docs/api-reference/chat/create

2.处理 Claude 的响应格式，保持和 OpenAI 输出一样的格式和字段，例如：

{
  "id": "cmpl-e5ca873642ca4f5d8b178c1742f9a8e8",
  "object": "chat.completion",
  "created": 1872961,
  "model": "moonshot-v1-128k",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "文案内容是关于一个名为“xxxx”的支付平台..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 498,
    "total_tokens": 509
  }
}

@hanxiantao
Copy link
Contributor

@CH3CHO 我的理解是有两个地方我还需要调整：

  1. Claude 和 OpenAI 请求体中的参数不一样，有些 Openai 有的请求体参数，在 Claude 里没有对应的参数，这种情况要怎么处理呢？比如说 OpenAI 有 n 参数，但是 Claude 里好像就没有。
    https://docs.anthropic.com/en/api/messages
    https://platform.openai.com/docs/api-reference/chat/create

2.处理 Claude 的响应格式，保持和 OpenAI 输出一样的格式和字段，例如：

{
  "id": "cmpl-e5ca873642ca4f5d8b178c1742f9a8e8",
  "object": "chat.completion",
  "created": 1872961,
  "model": "moonshot-v1-128k",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "文案内容是关于一个名为“xxxx”的支付平台..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 498,
    "total_tokens": 509
  }
}

我这边开发百度文心一言的时候，是参考的这个项目，有需要的话也可以参考下，https://github.com/songquanpeng/one-api/tree/91b80ae87945ed1a77b3507dd277ee9cdddaa0b4/relay/adaptor

@cr7258
Copy link
Contributor Author

cr7258 commented May 22, 2024

@hanxiantao 感谢，我研究一下

@cr7258 cr7258 requested a review from CH3CHO as a code owner June 3, 2024 03:52
@cr7258
adapt claude request and response
13829e3 
Signed-off-by: chengzw <chengzw258@163.com>
@cr7258
Copy link
Contributor Author

cr7258 commented Jun 3, 2024 •
edited

@CH3CHO PR 已经根据重新要求调整好了，请继续 review。测试的内容我更新在 PR 的描述里了

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 35.93%. Comparing base (6e4ade0) to head (490abc0).
Report is 5 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #969      +/-   ##
==========================================
- Coverage   35.97%   35.93%   -0.04%     
==========================================
  Files          69       69              
  Lines       11514    11536      +22     
==========================================
+ Hits         4142     4146       +4     
- Misses       7056     7074      +18     
  Partials      316      316

see 2 files with indirect coverage changes

@CH3CHO
Copy link
Collaborator

CH3CHO commented Jun 8, 2024

使用 OpenAI 非流式请求协议的相应格式是不是有点问题？看上去前面都没有 data: 前缀。

能参考下面两篇文档的内容用 NextChat 或者 LobeChat 测一下对接情况吗？

CH3CHO
CH3CHO reviewed Jun 11, 2024
View reviewed changes
plugins/wasm-go/extensions/ai-proxy/README.md
| `version` | string | 必填 | - | Claude 服务的 API 版本 |
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
|-----------|--------|------|-----|----------------------------------|
| `version` | string | 可选 | - | Claude 服务的 API 版本，默认为 2023-06-01 |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个字段建议在命名里就说明是 Anthropic Claude 用的。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CH3CHO CH3CHO CH3CHO left review comments

@johnlanni johnlanni Awaiting requested review from johnlanni johnlanni is a code owner

@WeixinX WeixinX Awaiting requested review from WeixinX WeixinX is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

AI 代理 Wasm 插件对接 Anthropic Claude
4 participants
@cr7258 @CH3CHO @hanxiantao @codecov-commenter