docker model launch llmfit

[ericcurtin]

Runs something like:

docker run --rm ghcr.io/alexsjones/llmfit

must ensure all the right holes are poked in the container, etc.

[ericcurtin]

Tagging @AlexsJones and @three-foxes-in-a-trenchcoat for awareness. Probably want to point at Docker Model Runner rather than Ollama also.

[sathiraumesh]

hi @ericcurtin can I look into this? I saw that llmfit has already included dmr as a provider. Maybe I can try to integrate it

[sathiraumesh]

Hi @ericcurtin @AlexsJones i was testing this localy. When
I set DOCKER_MODEL_RUNNER_HOST=http://model-runner.docker.internal and watched Docker model logs for incoming requests which means it uses the DMR backend.

docker run --rm -it \
  -e DOCKER_MODEL_RUNNER_HOST=http://model-runner.docker.internal \
  --entrypoint sh \
  ghcr.io/alexsjones/llmfit -c "llmfit"

In the default cli mode, it doesn't call the dmr backend, probably pointing to Ollama

docker run --rm \
  -e DOCKER_MODEL_RUNNER_HOST=http://model-runner.docker.internal \
  ghcr.io/alexsjones/llmfit recommend -n 3 

Should we wait for this to be available in default CLI mode in llmfit?

[AlexsJones]

Thanks for testing this @sathiraumesh! You're right — the DOCKER_MODEL_RUNNER_HOST env var was only being honored by llmfit's TUI mode; the CLI recommend subcommand wasn't probing any local provider at all.

Just cut a release that fixes this: v0.9.1. The recommend command now queries Ollama, MLX, llama.cpp, Docker Model Runner, and LM Studio — same as the TUI — so DMR will receive a request (and the installed status shows up in both text and JSON output).

Try the same command again with the new image and you should see the request land in Docker model logs.

[sathiraumesh]

Thanks @AlexsJones Ill test it

[sathiraumesh]

#837 @AlexsJones @ericcurtin you can review. I was thinking, should we allow interactive, maybe as another pr refrencing another issue

[AlexsJones]

Would docker run --rm -it ghcr.io/alexsjones/llmfit "" work to get the TUI going for now?
I am reluctant to create another image just for a custom CMD

[sathiraumesh]

@AlexsJones i will try to test. I think we probably need to change the existing implementation a bit to run interactively from DMR side. But I will test and see if there are any other changes that we might need

[sathiraumesh]

hi @AlexsJones i created a issue request here for the llmfit project to finalize the implementation here based on my findings and comments for this feature here. Can you have a look on this when you have time