(Checking how ASV benchmarks look without any code changes) - #1710
Draft
Sevans711 wants to merge 4 commits into
Draft
(Checking how ASV benchmarks look without any code changes)#1710Sevans711 wants to merge 4 commits into
Sevans711 wants to merge 4 commits into
Conversation
Collaborator
|
I did this too, actually. But an easier way to get the benchmarks running is to push your branch, then run ASV benchmarks directly in the Actions tab. That way you don't have to mess with a PR to get them running. |
ASV BenchmarkingBenchmark Comparison ResultsBenchmarks that have stayed the same:
|
Collaborator
Author
Ah, that makes sense! Does that lead to a nice comment with the benchmark results too? I suppose even if there isn't a comment it's definitely less noisy than making a draft PR |
7 tasks
Collaborator
|
Nah, you'd have to dig. I've been mostly looking at total time though, so for me it doesn't matter. |
Member
|
Leaving a note here in case it gets solved and disappears in the above benchmark run, we have been seeing a *1000 or so speedup on a test even when nothing changes, it appears as "no change". Very odd: 
This appears in this PR and others as well (at the time of writing). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not intended to be merged to main, just trying to check how ASV benchmarks look currently when the code is unchanged.
There's probably a better solution than having a draft PR for this, but this is the simplest/fastest way I could think of right now. I already attempted but struggled to generate ASV benchmark runs on my local machine with the same output format as what the run-benchmark label produces via github CI.
I originally created this with the intention of determining whether the reported benchmark performance degradation in #1705 is actually caused by the code changes there or not. Maybe this should be closed soon after that check, or maybe it should be open for a longer-term basis to repeat checks like this in the future? For example, it might have been useful to have something like this available for initially discovering #1605.