Improve ComponentMap / ComponentSet by jsiirola · Pull Request #3970 · Pyomo/pyomo

jsiirola · 2026-06-02T23:36:56Z

Fixes #755

Summary/Motivation:

This PR Updates COmponentMap / ComponentSet to prevent key collisions between ints and the id() of unhashable Components. The final solution proposed here is to store all keys coming from id() as a 2-tuple (combined with an internal class "flag"). This ended up being significantly more efficient than building a specialized HashKey class that would store the id() and not compare equal to int.

These changes make ComponentMap / ComponentSet slightly slower (~10-12%). To mitigate this, the PR includes two new containers, 'ObjectIdMap and ObjectIdSet. These containers are more efficient than the ComponentMap / ComponentSet, and approach the efficiency of a direct (manual) "dict-of-ids" implementation (see below).

As part of developing this PR, several other improvements were implemented:

to ComponentMap:
- return formal "view" objects from .keys(), .values(), and .items() to match the behavior of dict
- make KeyErrors raised here match those raised by dict
- simplify the string representation
- improve the robustness when comparing one ComponentMap to another
- make .update() API / behavior match dict
to ComponentSet:
- simplify the string representation
- improve the robustness when comparing one ComponentSet to another
- make .update() API / behavior match set
Update the ExpressionReplacementVisitor to officially accept ComponentMap for the substitution map

Performance data

Given the following test code:

m = ConcreteModel()
m.x = Var(range(100))
m.c = Constraint(range(100), rule=lambda m, i: m.x[i] >= i)

cm2 = ComponentMap()
cs2 = ComponentSet()
d2 = {}
s2 = set()
for _ in m.x.values():
    cm2[_] = 1
    cs2.add(_)
    d2[id(_)] = 1
    s2.add(id(_))
for _ in m.c.values():
    cm2[_] = 1
    cs2.add(_)
    d2[_] = 1
    s2.add(_)
for _ in range(100):
    cm2[_] = 1
    cs2.add(_)
    d2[_] = 1
    s2.add(_)

The original ComponentMap / ComponentSet implementation (from main), tested with 100k shots on Python 3.10 - 3.14:

    3.10    3.11    3.12    3.13    3.14   test
   3.536   2.685   2.867   2.782   2.611   'for _ in m.x.values():\n  cm[_] = _'
   3.379   2.224   2.360   2.291   2.158   'for _ in m.x.values():\n  cs.add(_)'
   1.725   1.424   1.546   1.502   1.491   'for _ in m.x.values():\n  d[id(_)] = _'
   2.029   1.711   1.794   1.707   1.697   'for _ in m.x.values():\n  d[id(_)] = (_,_)'
   1.827   1.405   1.555   1.503   1.511   'for _ in m.x.values():\n  s.add(id(_))'
   3.021   2.219   2.330   2.275   2.044   'for _ in m.c.values():\n  cm[_] = _'
   2.783   1.757   1.818   1.806   1.598   'for _ in m.c.values():\n  cs.add(_)'
   1.125   0.964   1.032   1.023   0.986   'for _ in m.c.values():\n  d[_] = _'
   1.227   0.948   1.010   0.992   0.979   'for _ in m.c.values():\n  s.add(_)'
   2.036   1.431   1.460   1.401   1.239   'for _ in range(100):\n  cm[_] = _'
   2.064   0.954   0.951   0.947   0.772   'for _ in range(100):\n  cs.add(_)'
   0.309   0.221   0.217   0.236   0.205   'for _ in range(100):\n  d[_] = _'
   0.387   0.203   0.191   0.212   0.204   'for _ in range(100):\n  s.add(_)'
   3.479   2.232   2.435   2.285   2.274   'for _ in m.x.values():\n  x = cm2[_]'
   1.662   1.395   1.508   1.467   1.472   'for _ in m.x.values():\n  x = d2[id(_)]'
   2.881   1.773   1.892   1.843   1.789   'for _ in m.c.values():\n  x = cm2[_]'
   1.111   0.977   1.021   1.037   0.981   'for _ in m.c.values():\n  x = d2[_]'
   1.993   1.066   1.047   1.033   0.861   'for _ in range(100):\n  x = cm2[_]'
   0.269   0.252   0.233   0.270   0.255   'for _ in range(100):\n  x = d2[_]'
   3.233   2.329   2.624   2.468   2.376   'for _ in m.x.values():\n  x = _ in cm2'
   3.185   2.360   2.616   2.462   2.348   'for _ in m.x.values():\n  x = _ in cs2'
   1.694   1.425   1.542   1.480   1.460   'for _ in m.x.values():\n  x = id(_) in d2'
   1.665   1.386   1.514   1.481   1.464   'for _ in m.x.values():\n  x = id(_) in s2'
   2.751   1.898   2.085   2.026   1.838   'for _ in m.c.values():\n  x = _ in cm2'
   2.661   1.896   2.081   2.022   1.821   'for _ in m.c.values():\n  x = _ in cs2'
   1.131   0.995   1.035   1.019   0.984   'for _ in m.c.values():\n  x = _ in d2'
   1.098   0.960   1.003   0.998   0.970   'for _ in m.c.values():\n  x = _ in s2'
   1.840   1.149   1.271   1.227   1.087   'for _ in range(100):\n  x = _ in cm2'
   1.823   1.156   1.277   1.221   1.141   'for _ in range(100):\n  x = _ in cs2'
   0.292   0.270   0.235   0.260   0.233   'for _ in range(100):\n  x = _ in d2'
   0.342   0.291   0.284   0.290   0.274   'for _ in range(100):\n  x = _ in s2'
   0.055   0.045   0.048   0.047   0.045   'for _ in cm.keys():\n  pass'
   0.054   0.045   0.047   0.046   0.045   'for _ in cm.values():\n  pass'
   0.013   0.009   0.011   0.011   0.012   'for _ in cs:\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.keys():\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.values():\n  pass'
   0.004   0.003   0.003   0.004   0.004   'for _ in s:\n  pass'

Results for the same tests on this PR:

    3.10    3.11    3.12    3.13    3.14   test
   3.968   3.140   3.238   3.165   2.937   'for _ in m.x.values():\n  cm[_] = _'
   3.771   2.693   2.740   2.707   2.467   'for _ in m.x.values():\n  cs.add(_)'
   1.664   1.444   1.548   1.505   1.494   'for _ in m.x.values():\n  d[id(_)] = _'
   1.981   1.733   1.802   1.692   1.706   'for _ in m.x.values():\n  d[id(_)] = (_,_)'
   1.816   1.424   1.581   1.460   1.516   'for _ in m.x.values():\n  s.add(id(_))'
   2.954   2.306   2.343   2.276   2.094   'for _ in m.c.values():\n  cm[_] = _'
   2.757   1.826   1.835   1.794   1.622   'for _ in m.c.values():\n  cs.add(_)'
   1.123   0.959   1.028   1.015   1.005   'for _ in m.c.values():\n  d[_] = _'
   1.219   0.948   1.015   0.989   0.987   'for _ in m.c.values():\n  s.add(_)'
   2.036   1.439   1.465   1.393   1.233   'for _ in range(100):\n  cm[_] = _'
   2.045   0.951   0.950   0.943   0.773   'for _ in range(100):\n  cs.add(_)'
   0.314   0.221   0.213   0.233   0.209   'for _ in range(100):\n  d[_] = _'
   0.398   0.203   0.191   0.210   0.207   'for _ in range(100):\n  s.add(_)'
   3.872   2.722   2.733   2.655   2.472   'for _ in m.x.values():\n  x = cm2[_]'
   1.619   1.408   1.499   1.457   1.481   'for _ in m.x.values():\n  x = d2[id(_)]'
   2.815   1.895   1.851   1.887   1.649   'for _ in m.c.values():\n  x = cm2[_]'
   1.109   0.983   1.013   1.022   1.008   'for _ in m.c.values():\n  x = d2[_]'
   1.991   1.115   1.070   0.959   0.863   'for _ in range(100):\n  x = cm2[_]'
   0.265   0.264   0.228   0.259   0.246   'for _ in range(100):\n  x = d2[_]'
   3.539   2.802   2.964   2.854   2.771   'for _ in m.x.values():\n  x = _ in cm2'
   3.615   2.818   2.955   2.858   2.702   'for _ in m.x.values():\n  x = _ in cs2'
   1.651   1.435   1.516   1.461   1.465   'for _ in m.x.values():\n  x = id(_) in d2'
   1.612   1.397   1.469   1.419   1.440   'for _ in m.x.values():\n  x = id(_) in s2'
   2.615   1.976   2.078   2.017   1.902   'for _ in m.c.values():\n  x = _ in cm2'
   2.603   1.979   2.081   2.018   1.865   'for _ in m.c.values():\n  x = _ in cs2'
   1.126   1.004   1.035   1.013   1.014   'for _ in m.c.values():\n  x = _ in d2'
   1.079   0.959   1.000   0.973   0.960   'for _ in m.c.values():\n  x = _ in s2'
   1.781   1.208   1.302   1.189   1.095   'for _ in range(100):\n  x = _ in cm2'
   1.777   1.209   1.270   1.190   1.091   'for _ in range(100):\n  x = _ in cs2'
   0.285   0.272   0.236   0.255   0.231   'for _ in range(100):\n  x = _ in d2'
   0.337   0.298   0.288   0.271   0.274   'for _ in range(100):\n  x = _ in s2'
   0.046   0.036   0.040   0.038   0.039   'for _ in cm.keys():\n  pass'
   0.047   0.037   0.041   0.039   0.040   'for _ in cm.values():\n  pass'
   0.013   0.009   0.011   0.011   0.012   'for _ in cs:\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.keys():\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.values():\n  pass'
   0.003   0.003   0.003   0.004   0.004   'for _ in s:\n  pass'

And if we use ObjectIdMap() / ObjectIdSet in lieu of the ComponentMap / ComponentSet:

    3.10    3.11    3.12    3.13    3.14   test
   2.553   2.189   2.321   2.222   2.169   'for _ in m.x.values():\n  cm[_] = _'
   2.207   1.718   1.818   1.748   1.778   'for _ in m.x.values():\n  cs.add(_)'
   1.680   1.431   1.546   1.503   1.489   'for _ in m.x.values():\n  d[id(_)] = _'
   2.000   1.727   1.787   1.705   1.695   'for _ in m.x.values():\n  d[id(_)] = (_,_)'
   1.815   1.420   1.525   1.480   1.505   'for _ in m.x.values():\n  s.add(id(_))'
   2.539   2.159   2.322   2.230   2.170   'for _ in m.c.values():\n  cm[_] = _'
   2.209   1.716   1.823   1.745   1.784   'for _ in m.c.values():\n  cs.add(_)'
   1.125   0.958   1.030   1.032   1.006   'for _ in m.c.values():\n  d[_] = _'
   1.223   0.959   1.013   0.993   0.994   'for _ in m.c.values():\n  s.add(_)'
   1.635   1.333   1.392   1.312   1.234   'for _ in range(100):\n  cm[_] = _'
   1.371   0.880   0.906   0.850   0.760   'for _ in range(100):\n  cs.add(_)'
   0.304   0.222   0.214   0.236   0.209   'for _ in range(100):\n  d[_] = _'
   0.382   0.203   0.194   0.211   0.208   'for _ in range(100):\n  s.add(_)'
   2.322   1.718   1.832   1.730   1.725   'for _ in m.x.values():\n  x = cm2[_]'
   1.640   1.403   1.498   1.462   1.475   'for _ in m.x.values():\n  x = d2[id(_)]'
   2.598   1.762   1.890   1.781   1.778   'for _ in m.c.values():\n  x = cm2[_]'
   1.252   0.974   1.027   1.030   0.997   'for _ in m.c.values():\n  x = d2[_]'
   1.490   0.914   0.957   0.929   0.889   'for _ in range(100):\n  x = cm2[_]'
   0.270   0.260   0.235   0.267   0.245   'for _ in range(100):\n  x = d2[_]'
   2.161   1.812   2.039   1.926   1.955   'for _ in m.x.values():\n  x = _ in cm2'
   2.190   1.829   2.023   1.926   1.910   'for _ in m.x.values():\n  x = _ in cs2'
   1.856   1.438   1.519   1.459   1.469   'for _ in m.x.values():\n  x = id(_) in d2'
   1.826   1.410   1.482   1.415   1.484   'for _ in m.x.values():\n  x = id(_) in s2'
   2.462   1.880   2.077   1.990   1.967   'for _ in m.c.values():\n  x = _ in cm2'
   2.413   1.851   2.078   1.989   1.959   'for _ in m.c.values():\n  x = _ in cs2'
   1.274   1.007   1.037   1.022   1.002   'for _ in m.c.values():\n  x = _ in d2'
   1.228   0.959   1.004   0.979   0.967   'for _ in m.c.values():\n  x = _ in s2'
   1.343   1.028   1.211   1.180   1.110   'for _ in range(100):\n  x = _ in cm2'
   1.342   1.038   1.203   1.177   1.090   'for _ in range(100):\n  x = _ in cs2'
   0.297   0.277   0.239   0.258   0.236   'for _ in range(100):\n  x = _ in d2'
   0.335   0.294   0.287   0.273   0.277   'for _ in range(100):\n  x = _ in s2'
   0.046   0.036   0.040   0.040   0.040   'for _ in cm.keys():\n  pass'
   0.046   0.036   0.040   0.039   0.039   'for _ in cm.values():\n  pass'
   0.013   0.009   0.011   0.011   0.012   'for _ in cs:\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.keys():\n  pass'
   0.006   0.005   0.006   0.006   0.006   'for _ in d.values():\n  pass'
   0.003   0.003   0.003   0.004   0.004   'for _ in s:\n  pass'

Changes proposed in this PR:

(see above)

AI-Use Disclosure

AI tools were NOT used during the preparation of this PR

or

Notes for reviewers (optional):

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

…n necessary

…nt is not callable

…tVisitor

codecov · 2026-06-03T00:40:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.15%. Comparing base (93f71a7) to head (0149557).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3970      +/-   ##
==========================================
+ Coverage   90.13%   90.15%   +0.01%     
==========================================
  Files         917      917              
  Lines      109086   109201     +115     
==========================================
+ Hits        98329    98448     +119     
+ Misses      10757    10753       -4

Flag	Coverage Δ
builders	`29.13% <52.17%> (+0.04%)`	⬆️
default	`86.14% <100.00%> (?)`
expensive	`35.10% <57.76%> (?)`
linux	`87.67% <100.00%> (-1.97%)`	⬇️
linux_other	`87.67% <100.00%> (+0.01%)`	⬆️
oldsolvers	`28.05% <52.17%> (+0.02%)`	⬆️
osx	`83.13% <100.00%> (+0.01%)`	⬆️
win	`85.44% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

michaelbynum · 2026-08-04T19:01:14Z

-        x1 = var_map[id(m.x1)]
-        x2 = var_map[id(m.x2)]
-        x3 = var_map[id(m.x3)]
-        y1 = var_map[id(m.y1)]
-        y2 = var_map[id(m.y2)]
-        y3 = var_map[id(m.y3)]
-        z1 = var_map[id(m.z1)]
+        x1 = var_map[m.x1]
+        x2 = var_map[m.x2]
+        x3 = var_map[m.x3]
+        y1 = var_map[m.y1]
+        y2 = var_map[m.y2]
+        y3 = var_map[m.y3]
+        z1 = var_map[m.z1]


I don't understand why these needed changed without changing the gurobi-minlp visitor.

These tests were exploiting a bug in ComponentMap where a VarData could be looked up by either the object or it's integer id() (i.e., we were not handling a hash collision correctly). This PR resolves that bug, so these tests needed to be updated.

jsiirola added 30 commits February 8, 2026 13:26

Return HashKey fro HashDispatcher (and not id())

c6389b8

Improve tuple handling in hash dispatcher: only generate HashKeys whe…

6e10267

…n necessary

Minor cleanup

ad99ee0

Make HashDispatcher callable (support use in place of id())

6f793d9

Minor ComponentMap performance improvements

6ba2426

Support deriving from ComponentMap/ComponentSet

59f7ba2

Standardize str() and KeyError for COmponentMap/Set

fe4d4da

Add ObjectIdMap/ObjectIdSet

c2238d3

Minor code cleanup/fixes

e60cabb

NFC: update comments/docstrings

e3c8cbd

Support cleaner fallback to ComponentMap when first positional argume…

9b3e036

…nt is not callable

Add / update tests

bfd5092

Merge branch 'main' into component-map-hashing

4cc18e8

NFC: apply black

a865f03

Avoid resolving expressions created during ComponentMap comparisons

c8823a7

Generate more human-readable str(ComponentMap)/str(COmponentSet)

dcee582

ComponetMap keys/values/items should return views

156557a

Merge branch 'main' into component-map-hashing

5458af6

Merge branch 'main' into component-map-hashing

bd7835e

Improve/test HashKey repr/str

ad1fa4f

NFC: update docs

facfa68

Make update() API consistent with dict/set

7560771

Fix typo

1a4fecc

We don't need a full HashKey class; just a flag to add in a native tuple

5c1dd71

Support ComponentMap as the substitution map for ExpressionReplacemen…

484b6f7

…tVisitor

Don't look up ComponentMap values by key id()

11cc469

Track updated exception message

7ee41f0

Catch that comparing numpy types can produce numpy bools

c455f6b

Guard against object collection under ExpressionReplacementVisitor

9365fff

Update to track simplification of string representations

295d8b6

Expose ObjectIdMap / ObjectIdSet through pyomo.common.collections

a6ac21f

Update test for pypy

5ebac4b

blnicho requested a review from michaelbynum June 16, 2026 18:49

pyomo-autotest added the AT: STALE label Jun 23, 2026

Merge branch 'main' into component-map-hashing

0149557

pyomo-autotest removed the AT: STALE label Jun 26, 2026

pyomo-autotest added the AT: STALE label Jul 16, 2026

michaelbynum approved these changes Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

Improve ComponentMap / ComponentSet - #3970

Improve ComponentMap / ComponentSet#3970
jsiirola wants to merge 33 commits into
Pyomo:mainfrom
jsiirola:component-map-hashing

jsiirola commented Jun 2, 2026

Uh oh!

codecov Bot commented Jun 3, 2026 •

edited

Loading

Uh oh!

michaelbynum Aug 4, 2026

Uh oh!

jsiirola Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jsiirola commented Jun 2, 2026

Fixes #755

Summary/Motivation:

Performance data

Changes proposed in this PR:

AI-Use Disclosure

Legal Acknowledgement

Uh oh!

codecov Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

michaelbynum Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

jsiirola Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Jun 3, 2026 •

edited

Loading