LibreChat Update Causes 500 Error With Azure Inception Jais Model

by gitunigon 66 views
Iklan Headers

Hey everyone, we've got a bit of a situation on our hands with the LibreChat update and the Azure Inception Jais model. It seems like a recent update has introduced a bug that's causing chat completions to fail. Let's dive into the details and figure out what's going on.

The Problem: 500 Error with Jais Model

So, the main issue we're facing is a 500 error when trying to use the Inception Jais model from Azure through LiteLLM. For those not familiar, the Inception Jais model is specifically designed for Arabic language processing, and we've been relying on it for our chat completions. However, after a recent LibreChat update, things have gone a bit haywire. This error occurs when attempting to use the Jais model for chat completions, rendering the model unusable within LibreChat. This directly impacts users who require Arabic language support, as the Jais model is specifically tailored for this purpose. The 500 error suggests a server-side issue, indicating that the problem lies within the interaction between LibreChat, LiteLLM, and the Azure Foundry endpoint. The error messages point to a TypeError: unhashable type: 'dict', which suggests a data structure issue during the language detection process. This means that the system is encountering a Python dictionary where it expects a hashable type, such as a string or tuple. Understanding the root cause requires a deep dive into the code changes between the identified commits and how they affect the data flow between LibreChat and the Azure service. Furthermore, the error message includes a stack trace that helps pinpoint the exact location in the code where the error occurs, aiding developers in identifying the problematic section. The error arises during the language detection phase, specifically within the detect_language function in the billingproxy/apis/jais/protocol.py file, indicating that the input to this function is not in the expected format. This could be due to a change in how messages are structured or how content is extracted from them. The traceback also reveals that the issue is triggered when attempting to create a set from the words in the message content, which expects each word to be a hashable type. This further narrows down the problem to the structure of the message content being passed to the detect_language function. To resolve this, developers need to examine the code changes that occurred between the identified commits, focusing on how message content is processed and passed to the language detection function. This may involve changes in data serialization, message formatting, or data type handling. The resolution may also require modifications to the detect_language function to handle the new data structure correctly or to perform necessary data type conversions. Collaboration between the LibreChat and LiteLLM teams may be necessary to ensure compatibility between the two systems and to identify the root cause of the error definitively. The error impacts the reliability and usability of LibreChat for users who depend on the Jais model for Arabic language interactions, necessitating a swift resolution. The error message's detail allows for targeted debugging, reducing the time needed to identify and fix the issue. The error message also provides valuable insights into the internal workings of the system, which can help prevent similar issues in the future. The consistent occurrence of the error across all chat completions with the Jais model highlights the need for thorough testing and quality assurance processes during software updates and integrations. The issue underscores the importance of understanding the data flow between different components in a complex system and the impact that changes in one component can have on others.

The Culprit: LibreChat Update

We've narrowed down the problematic changes to a specific window in LibreChat's commit history:

This means that whatever change introduced the bug happened somewhere between these two commits. This provides a specific timeframe for developers to examine the codebase changes, significantly reducing the scope of the investigation and making it easier to pinpoint the root cause. This method of identifying a specific range of commits is a common and effective strategy for bug hunting, especially in large codebases with frequent updates. By focusing on the changes within this timeframe, developers can systematically review the code modifications, focusing on areas that are likely to affect the interaction with the Azure Inception Jais model. This narrowed scope is essential because it allows for a more detailed analysis of the code, increasing the chances of identifying the exact change that introduced the bug. The commit range also helps in identifying the specific developers who worked on the code during this period, facilitating direct communication and knowledge sharing. This is particularly useful if the bug is related to a subtle change in the code logic or data structures that might not be immediately obvious from the commit messages. The practice of narrowing down the potential causes to a specific timeframe is not only applicable to software development but can also be used in various fields where changes over time need to be analyzed to identify a problem's origin. In the context of LibreChat, this approach demonstrates the importance of maintaining a detailed version history and using it effectively to diagnose and resolve issues. The identified commit range acts as a starting point for a deeper investigation, which may involve code reviews, debugging sessions, and potentially reverting to an earlier version of the code to confirm the issue and test potential fixes. The successful identification of this range highlights the importance of a well-organized development process and the use of version control systems in managing software projects. It also underscores the value of clear and concise commit messages, which can help developers understand the purpose and impact of each change, making it easier to diagnose and resolve issues that arise later.

Configuration Details

To give you a clearer picture, here’s our LiteLLM configuration for Jais:

  - model_name: "[Inception] جبل جس (Jais 30B Chat)"
    litellm_params:
      model: azure/jais-30b-chat
      api_base: https://-----.openai.azure.com/
      api_version: "2023-12-01-preview"
    model_info:
      max_tokens: 7500
      max_input_tokens: 7500

And this is from our librechat.yaml for Jais:

    - name: "Inception"
      apiKey: "-----"
      baseURL: "https://litellm.river.berkeley.edu"
      iconURL: "https://-----/jais.png"
      models:
        default: ["[Inception] جبل جس (Jais 30B Chat)"]
        fetch: false
      maxContextTokens: 8000
      titleConvo: true
      titleModel: "gpt-3.5-turbo"
      summarize: true
      summaryModel: "gpt-3.5-turbo"
      forcePrompt: false
      modelDisplayLabel: "Inception"

These configuration details are crucial for understanding how the Jais model is integrated into LibreChat through LiteLLM, providing a complete picture of the setup. These snippets show how the model is defined within both LiteLLM and LibreChat, including important parameters such as the model name, API base, API version, and token limits. Sharing the configurations enables developers to reproduce the issue in a controlled environment, which is essential for effective debugging and testing. The LiteLLM configuration specifies the connection details for accessing the Azure OpenAI service, including the model identifier (azure/jais-30b-chat) and the API endpoint (api_base). This information is necessary for establishing a connection to the Jais model and sending requests for chat completions. The model_info section defines the maximum number of tokens that can be used for both input and output, which is important for managing resource consumption and ensuring the model operates within its limits. The LibreChat configuration, on the other hand, defines how the Jais model is exposed within the LibreChat interface. It includes the model name (`