how you scaled up the parameters?
#1
by
jiyintor
- opened
Could you introduce how you scaled up the parameters? Did you achieve this by adding more experts?
I'd love to know the answer to this also
The simple answer here is in this model's config.json:
"num_hidden_layers": 36,
versus the stock"num_hidden_layers": 24,
But I'm curious too. Are these layers duplicates or has there been finetuning?
I'd be keen to see a spin-off of this model with an expert pruning to reduce the model's size back down, but with less width, more depth.