Correct promt format is crucial.
At first, when i tried this model with my usual reasoning questions, it failed a lot (wrong answers). And only when i used this >>> llama-cli -m Phi-3-medium-128k-instruct-Q4_K_M.gguf -p " " --in-prefix "<|user|>\n" --in-suffix "<|end|>\n<|assistant|>\n" , it suddenly became much smarter :)
I wish every model had correct settings for llama.cpp. Every time i struggle with it...
it should already be set to the right one
though for my curiousity, can you try tokenizing your prompt? the \n should get removed (which I think is a problem personally)
it should already be set to the right one
I'm not good at this. Maybe you could provide the correct settings / promt format for llama.cpp...
ollama modelfile:
FROM /your/path/Phi-3-medium-128k-instruct-Q8_0.gguf
PARAMETER stop "<|assistant|>"
PARAMETER stop "<|end|>"
PARAMETER stop "<|user|>"
PARAMETER temperature 0.3
TEMPLATE """{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}<|end|>
"""