Skip to content

基于TGI的模型server无法在v100上进行推理加速 #45

Open
Open
基于TGI的模型server无法在v100上进行推理加速#45

Description

@BruceMa29
BruceMa29
opened on Nov 2, 2023

@ruixie 您好，由于v100无法使用FlashAttention，所以是不是也没办法在v100上对模型服务进行推理加速呢？

Activity

  1. ruixie commented on Nov 7, 2023

    @ruixie
    ruixie
    on Nov 7, 2023
    Collaborator

    你好，TGI的推理加速很大部分是依赖FlashAttention的，所以在V100上加速效果确实会受到影响

  2. BruceMa29 commented on Nov 7, 2023

    @BruceMa29
    BruceMa29
    on Nov 7, 2023
    Author

    @ruixie 好的，因为FlashAttention在v100上跑不起来，所以那也就没法进行加速了。原生transformers压测的吞吐量很低

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions