Rrmove memory estimation logic and related structures

[ilopezluna]

This PR removes the memory estimation and tracking logic from the model runner, simplifying the codebase.
We decided to remove it as we fail to estimate in many cases.

[gemini-code-assist]

Summary of Changes

Hello @ilopezluna, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly simplifies the model-runner's operational framework by entirely eliminating the memory estimation and management logic. This change removes the complexity associated with calculating and tracking system and model memory usage, as well as the runtime checks that previously occurred during model loading and eviction. The primary impact is a more straightforward and less memory-aware scheduling mechanism, which now focuses on managing model instances based on available slots rather than detailed memory footprints.

Highlights

• Memory Estimation Logic Removed: The entire memory estimation and management subsystem, including components for gathering system memory information (RAM, VRAM) and checking model memory requirements, has been completely removed from the codebase.
• Simplified Model Loading and Eviction: The logic for loading and evicting models within the scheduler has been streamlined. It no longer considers memory constraints, relying solely on the availability of runner slots.
• Codebase Cleanup: The dedicated pkg/inference/memory package, along with all related imports, struct fields, function parameters, and method calls across main.go, http_handler.go, loader.go, and scheduler.go, has been deleted or refactored out.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request removes the memory estimation logic and related data structures. The changes are consistently applied across the codebase, resulting in a significant simplification. The model runner scheduling logic in pkg/inference/scheduling/loader.go is now based on available slots instead of memory. This removes the previous safeguards against loading models that might exceed system memory capacity, which is a significant change in behavior. The implementation of this removal is clean and well-executed.

As a minor cleanup suggestion, the formatMemorySize function in pkg/inference/scheduling/loader.go and its test TestFormatMemorySize in pkg/inference/scheduling/loader_test.go appear to be no longer used and could be removed in a follow-up change.

[sourcery-ai]

Hey there - I've reviewed your changes - here's some feedback:

• Now that all runtime memory estimation has been removed, the IgnoreRuntimeMemoryCheck field in the model create request appears to be unused; consider removing it or clearly marking it deprecated to avoid confusion for API consumers.
• With the memory tracking logic removed from loader, helpers like formatMemorySize are no longer used in production code (only tests); consider inlining/removing them or relocating to a more generic util package if you still want to keep the tests around.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- Now that all runtime memory estimation has been removed, the `IgnoreRuntimeMemoryCheck` field in the model create request appears to be unused; consider removing it or clearly marking it deprecated to avoid confusion for API consumers.
- With the memory tracking logic removed from `loader`, helpers like `formatMemorySize` are no longer used in production code (only tests); consider inlining/removing them or relocating to a more generic util package if you still want to keep the tests around.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}