Repository navigation
landice/ismip6_forcing/atmosphere fails with an overflow error #919
Description
The landice/ismip6_forcing/atmosphere fails with an overflow error (full traceback at the end of this), with the relevant lines being:
File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/landice/tests/ismip6_forcing/atmosphere/process_smb.py", line 154, in run
write_netcdf(input_file_combined, combined_file_temp)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
and
OverflowError: Python integer 5670752832 out of bounds for int32
The only int32 variable in the dataset being written is year. I tested with input_file_combined.drop_vars(['year']) before the call to write_netcdf and it failed with the same error, so apparently that is not the issue. So, it is unclear what variable is causing the error.
Traceback (most recent call last):
File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/run/serial.py", line 322, in _log_and_run_test
_run_test(test_case, available_resources)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/run/serial.py", line 419, in _run_test
_run_step(test_case, step, test_case.new_step_log_file,
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
available_resources)
^^^^^^^^^^^^^^^^^^^^
File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/run/serial.py", line 470, in _run_step
step.run()
~~~~~~~~^^
File "/global/cfs/cdirs/m4288/users/trhille/compass/compass/landice/tests/ismip6_forcing/atmosphere/process_smb.py", line 154, in run
write_netcdf(input_file_combined, combined_file_temp)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/mpas_tools/io.py", line 156, in write_netcdf
ds.to_netcdf(
~~~~~~~~~~~~^
out_filename, encoding=encodingDict, format=format, engine=engine
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/core/dataset.py", line 2109, in to_netcdf
return to_netcdf( # type: ignore[return-value] # mypy cannot resolve the overloads:(
self,
...<10 lines>...
auto_complex=auto_complex,
)
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/backends/api.py", line 2100, in to_netcdf
store.close()
~~~~~~~~~~~^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/backends/scipy_.py", line 298, in close
self._manager.close()
~~~~~~~~~~~~~~~~~~~^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/xarray/backends/file_manager.py", line 234, in close
file.close()
~~~~~~~~~~^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 298, in close
self.flush()
~~~~~~~~~~^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 411, in flush
self._write()
~~~~~~~~~~~^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 423, in _write
self._write_var_array()
~~~~~~~~~~~~~~~~~~~~~^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 471, in _write_var_array
self._write_var_metadata(name)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 510, in _write_var_metadata
self._pack_int(vsize)
~~~~~~~~~~~~~~^^^^^^^
File "/global/cfs/cdirs/fanssie/users/trhille/miniforge3/envs/dev_compass_1.8.0-alpha.2/lib/python3.13/site-packages/scipy/io/_netcdf.py", line 785, in _pack_int
self.fp.write(array(value, '>i').tobytes())
~~~~~^^^^^^^^^^^^^
OverflowError: Python integer 5670752832 out of bounds for int32
Activity
xylar commentedon Aug 21, 2025
@trhille, are there int64 variables? MPAS doesn't support that format so they get converted to int32 in
write_netcdf. That could be the issue. This is a relatively recent change inwrite_netcdf()but previouslyint64variables were incorrectly getting read in with a different format (I think float) in MPAS and they now produce an error.trhille commentedon Aug 21, 2025
@xylar, no it looks like everything else is float32 or float64.
xylar commentedon Aug 24, 2025
If you change
write_netcdf()to justinput_file_combined.to_netcdf(), does that work? If so, can you look at the result and see which field might be the culprit?trhille commentedon Sep 3, 2025
@xylar, that works! Is there a reason to keep
write_netcdf(), or can we just change it to usingto_netcdf()?xylar commentedon Sep 3, 2025
@trhille, in general, you should prefer
write_netcdf()whenever the output will be used by MPAS components because it does a lot of important checks to make sure things are compatible. I would take a look at the results formto_netcdf()and try to find out what was causing trouble with regard toint64if you can. If MALI actually can read the files, it may not be for a good reason since MPAS can't actually readint64fields.