Alibaba Hosted Model plz
Hi,
Since this project comes from Alibaba, any chance you could get them to host the Tongyi-DeepResearch-30B-A3B model on Alibaba Cloud?
This project's inference code is set up to be locally hosted with 8 (!) vLLM instances. That is a non-trivial hardware requirement.
I re-wrote the inference code to use an OpenAPI compatible endpoint, then set up Alibaba-NLP/Tongyi-DeepResearch-30B-A3B as a "Deployment" with the DeepInfra company.
It would be much easier if the model was just available from Alibaba. Perhaps Alibaba could add an "Experimental" section to their models, to make it clear that this may not yet (?) be a full production model.
Thanks for your consideration,
-Jeff
According to the update README.md in the project's Github, the model on OpenRouter is now supported with a few lines of code changes:
https://openrouter.ai/alibaba/tongyi-deepresearch-30b-a3b
I am running with some different code than the git repo, but I have found the OpenRouter model to have some issues compared to the DeepInfra Deployment I did. The latter, I believe, uses the Tongyi-DeepResearch-30B-A3B model config/template from Huggingface when it imports the model. I don't know how OpenRouter sets it up, but they don't appear to be working exactly the same. I haven't dug deep into the issue, but FYI, others may see the same.
The OpenRouter model is being directed to using Atlas Cloud.
Unfortunately, it is hitting 429 rate limits. I did very few queries with it (I am mostly using DeepInfra). The limits appear to be an issue with Atlas Cloud itself.
I used Atlas Cloud's API directly with an Atlas Cloud account, and hit the same "429" rate limit on the first query. So it seems Atlas Cloud can't handle serving this model or is having some issue.
OpenRouter API URL: https://openrouter.ai/api/v1
OpenRouter Model Name: alibaba/tongyi-deepresearch-30b-a3b
Atlas Cloud API URL:https://api.atlascloud.ai/v1
Atlast Cloud Model Name: Alibaba-NLP/Tongyi-DeepResearch-30B-A3B
Please add the model to Alibaba Cloud!