Fix stencils over-transfer of neighbor data by jpsamaroo · Pull Request #737 · JuliaParallel/Dagger.jl

jpsamaroo · 2026-08-13T21:04:33Z

This is a regression introduced by prior stencil optimizations, which improved multi-threaded execution but significantly regressed multi-process and multi-rank.

Written by Claude Opus

…on_dag

…nt spaces

github-actions · 2026-08-13T21:25:37Z

Dagger benchmarks: `dirty` vs `master`

Multi-threaded benchmarks (4 threads)

Dagger benchmarks: `dirty` vs `master`

	master	dirty	master / dirty
array/dagger/N=1024 (block 128)/add (X + X)	20.8 ± 3.6 ms	21.8 ± 2.8 ms	0.955 ± 0.21
array/dagger/N=1024 (block 128)/alloc (rand)	16 ± 1.9 ms	15.7 ± 0.76 ms	1.02 ± 0.13
array/dagger/N=1024 (block 128)/broadcast (X .+ 1)	14.9 ± 0.86 ms	16 ± 1.9 ms	0.929 ± 0.12
array/dagger/N=1024 (block 128)/map (sin.(X))	15.3 ± 2.4 ms	14.5 ± 0.5 ms	1.06 ± 0.17
array/dagger/N=1024 (block 128)/norm	16.1 ± 0.44 ms	16.2 ± 2 ms	0.994 ± 0.12
array/dagger/N=1024 (block 128)/reduce (sum)	0.0325 ± 0.0029 s	0.034 ± 0.0039 s	0.954 ± 0.14
array/dagger/N=1024 (block 128)/transpose (permutedims)	13.5 ± 0.67 ms	12.4 ± 1.7 ms	1.08 ± 0.15
array/dagger/N=1024 (block 512)/add (X + X)	3.63 ± 0.26 ms	4.32 ± 0.98 ms	0.84 ± 0.2
array/dagger/N=1024 (block 512)/alloc (rand)	3.4 ± 0.66 ms	3.09 ± 0.54 ms	1.1 ± 0.29
array/dagger/N=1024 (block 512)/broadcast (X .+ 1)	2.81 ± 0.29 ms	2.6 ± 0.097 ms	1.08 ± 0.12
array/dagger/N=1024 (block 512)/map (sin.(X))	7.01 ± 1.8 ms	7.25 ± 0.11 ms	0.966 ± 0.24
array/dagger/N=1024 (block 512)/norm	1.37 ± 1.7 ms	1.39 ± 1 ms	0.983 ± 1.5
array/dagger/N=1024 (block 512)/reduce (sum)	2.49 ± 0.066 ms	2.36 ± 0.058 ms	1.05 ± 0.038
array/dagger/N=1024 (block 512)/transpose (permutedims)	6.99 ± 0.3 ms	6.52 ± 0.34 ms	1.07 ± 0.072
array/dagger/N=256 (block 128)/add (X + X)	3.34 ± 1.5 ms	2.21 ± 0.17 ms	1.51 ± 0.7
array/dagger/N=256 (block 128)/alloc (rand)	1.61 ± 0.086 ms	1.55 ± 0.72 ms	1.04 ± 0.48
array/dagger/N=256 (block 128)/broadcast (X .+ 1)	1.53 ± 1.6 ms	1.53 ± 0.66 ms	0.998 ± 1.2
array/dagger/N=256 (block 128)/map (sin.(X))	1.32 ± 0.62 ms	1.26 ± 0.039 ms	1.04 ± 0.49
array/dagger/N=256 (block 128)/norm	1.74 ± 0.86 ms	1.43 ± 0.95 ms	1.22 ± 1
array/dagger/N=256 (block 128)/reduce (sum)	4.43 ± 0.88 ms	5.55 ± 4 ms	0.798 ± 0.59
array/dagger/N=256 (block 128)/transpose (permutedims)	1.75 ± 1.3 ms	1.67 ± 0.16 ms	1.05 ± 0.82
array/dagger/N=256 (block 256)/add (X + X)	1.05 ± 0.071 ms	0.978 ± 0.1 ms	1.07 ± 0.13
array/dagger/N=256 (block 256)/alloc (rand)	0.935 ± 0.16 ms	1.09 ± 0.17 ms	0.858 ± 0.2
array/dagger/N=256 (block 256)/broadcast (X .+ 1)	0.622 ± 0.067 ms	0.477 ± 0.014 ms	1.31 ± 0.15
array/dagger/N=256 (block 256)/map (sin.(X))	1.2 ± 0.13 ms	1.04 ± 0.036 ms	1.15 ± 0.13
array/dagger/N=256 (block 256)/norm	0.553 ± 0.14 ms	0.502 ± 0.013 ms	1.1 ± 0.29
array/dagger/N=256 (block 256)/reduce (sum)	0.919 ± 0.15 ms	0.888 ± 0.07 ms	1.04 ± 0.19
array/dagger/N=256 (block 256)/transpose (permutedims)	1.27 ± 0.32 ms	1.02 ± 0.24 ms	1.24 ± 0.42
linalg/dagger/N=1024 (block 128)/cholesky	0.0985 ± 0.0041 s	0.0884 ± 0.025 s	1.11 ± 0.32
linalg/dagger/N=1024 (block 128)/lu	0.203 ± 0.022 s	0.201 ± 0.015 s	1.01 ± 0.13
linalg/dagger/N=1024 (block 128)/matmul (A*A)	0.241 ± 0.037 s	0.238 ± 0.0086 s	1.01 ± 0.16
linalg/dagger/N=1024 (block 128)/matvec (A*x)	28.6 ± 2.4 ms	27.7 ± 4 ms	1.03 ± 0.17
linalg/dagger/N=1024 (block 128)/qr	0.166 ± 0.0051 s	0.179 ± 0.019 s	0.929 ± 0.1
linalg/dagger/N=1024 (block 128)/solve (A\b via lu)	0.306 ± 0.012 s	0.297 ± 0.035 s	1.03 ± 0.13
linalg/dagger/N=1024 (block 128)/svd	28.2 s	27.8 s	1.01
linalg/dagger/N=1024 (block 128)/syrk (A'*A)	0.147 ± 0.01 s	0.159 ± 0.011 s	0.924 ± 0.088
linalg/dagger/N=1024 (block 512)/cholesky	24.6 ± 2.2 ms	25.5 ± 13 ms	0.964 ± 0.49
linalg/dagger/N=1024 (block 512)/lu	0.042 ± 0.0029 s	0.0453 ± 0.0037 s	0.927 ± 0.099
linalg/dagger/N=1024 (block 512)/matmul (A*A)	0.0487 ± 0.0082 s	0.0462 ± 0.0085 s	1.06 ± 0.26
linalg/dagger/N=1024 (block 512)/matvec (A*x)	3.39 ± 0.47 ms	3.03 ± 1 ms	1.12 ± 0.42
linalg/dagger/N=1024 (block 512)/qr	0.111 ± 0.0035 s	0.114 ± 0.0064 s	0.968 ± 0.062
linalg/dagger/N=1024 (block 512)/solve (A\b via lu)	0.0515 ± 0.0035 s	0.0507 ± 0.01 s	1.02 ± 0.22
linalg/dagger/N=1024 (block 512)/svd	0.0353 h	0.0362 h	0.976
linalg/dagger/N=1024 (block 512)/syrk (A'*A)	0.038 ± 0.0035 s	0.0348 ± 0.0043 s	1.09 ± 0.17
linalg/dagger/N=256 (block 128)/cholesky	6.76 ± 2.4 ms	7.82 ± 1.8 ms	0.865 ± 0.37
linalg/dagger/N=256 (block 128)/lu	10.1 ± 2.6 ms	12.8 ± 4.2 ms	0.79 ± 0.33
linalg/dagger/N=256 (block 128)/matmul (A*A)	6.2 ± 4.7 ms	5.19 ± 1.9 ms	1.19 ± 1
linalg/dagger/N=256 (block 128)/matvec (A*x)	3.03 ± 0.38 ms	2.77 ± 1.3 ms	1.09 ± 0.52
linalg/dagger/N=256 (block 128)/qr	9.18 ± 1.9 ms	10.2 ± 1.3 ms	0.903 ± 0.22
linalg/dagger/N=256 (block 128)/solve (A\b via lu)	18.9 ± 3.3 ms	18.4 ± 1.8 ms	1.03 ± 0.21
linalg/dagger/N=256 (block 128)/svd	0.674 ± 0.018 s	0.686 ± 0.029 s	0.982 ± 0.049
linalg/dagger/N=256 (block 128)/syrk (A'*A)	6.51 ± 1.2 ms	7.03 ± 4.9 ms	0.926 ± 0.67
linalg/dagger/N=256 (block 256)/cholesky	2.95 ± 0.2 ms	5.38 ± 2.6 ms	0.549 ± 0.27
linalg/dagger/N=256 (block 256)/lu	4.24 ± 0.5 ms	4.14 ± 0.16 ms	1.02 ± 0.13
linalg/dagger/N=256 (block 256)/matmul (A*A)	2.55 ± 0.51 ms	2.45 ± 0.59 ms	1.04 ± 0.33
linalg/dagger/N=256 (block 256)/matvec (A*x)	1.28 ± 0.39 ms	1.26 ± 0.12 ms	1.02 ± 0.33
linalg/dagger/N=256 (block 256)/qr	5.45 ± 0.34 ms	4.93 ± 0.26 ms	1.11 ± 0.091
linalg/dagger/N=256 (block 256)/solve (A\b via lu)	9.45 ± 1.6 ms	8.9 ± 2.5 ms	1.06 ± 0.35
linalg/dagger/N=256 (block 256)/svd	0.628 ± 0.041 s	0.509 ± 0.021 s	1.23 ± 0.096
linalg/dagger/N=256 (block 256)/syrk (A'*A)	3.15 ± 0.81 ms	4.04 ± 1 ms	0.78 ± 0.28
stencil/dagger/N=1024 (block 128)/alloc (neighbors Wrap)	30.6 ± 1.8 ms	0.0316 ± 0.0008 s	0.968 ± 0.062
stencil/dagger/N=1024 (block 128)/assign (const)	12.3 ± 1.8 ms	11.6 ± 0.87 ms	1.06 ± 0.17
stencil/dagger/N=1024 (block 128)/multi-expr	26.4 ± 1.7 ms	24.4 ± 0.2 ms	1.08 ± 0.07
stencil/dagger/N=1024 (block 128)/neighbors (Clamp)	25.5 ± 1.3 ms	24.8 ± 1.1 ms	1.03 ± 0.068
stencil/dagger/N=1024 (block 128)/neighbors (Pad)	23.4 ± 2.1 ms	25.6 ± 0.36 ms	0.913 ± 0.083
stencil/dagger/N=1024 (block 128)/neighbors (Reflect)	24.5 ± 0.75 ms	25.6 ± 1.4 ms	0.955 ± 0.059
stencil/dagger/N=1024 (block 128)/neighbors (Wrap)	23.3 ± 1.8 ms	25.6 ± 1.1 ms	0.911 ± 0.081
stencil/dagger/N=1024 (block 128)/update (+)	14.7 ± 0.6 ms	15.2 ± 1.5 ms	0.964 ± 0.1
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap)	9.7 ± 2 ms	10.4 ± 1.1 ms	0.933 ± 0.22
stencil/dagger/N=1024 (block 512)/assign (const)	1.35 ± 0.03 ms	1.5 ± 0.35 ms	0.901 ± 0.21
stencil/dagger/N=1024 (block 512)/multi-expr	3.7 ± 1.5 ms	3.1 ± 0.011 ms	1.19 ± 0.49
stencil/dagger/N=1024 (block 512)/neighbors (Clamp)	6.9 ± 0.45 ms	7.23 ± 1.4 ms	0.955 ± 0.2
stencil/dagger/N=1024 (block 512)/neighbors (Pad)	8.01 ± 1.4 ms	8 ± 1.1 ms	1 ± 0.23
stencil/dagger/N=1024 (block 512)/neighbors (Reflect)	6.85 ± 0.63 ms	7.95 ± 1.6 ms	0.862 ± 0.19
stencil/dagger/N=1024 (block 512)/neighbors (Wrap)	6.16 ± 0.4 ms	8.17 ± 0.79 ms	0.754 ± 0.088
stencil/dagger/N=1024 (block 512)/update (+)	2.64 ± 0.6 ms	2.26 ± 2.4 ms	1.17 ± 1.2
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap)	3.06 ± 0.13 ms	2.94 ± 0.48 ms	1.04 ± 0.17
stencil/dagger/N=256 (block 128)/assign (const)	1.85 ± 0.13 ms	1.3 ± 0.43 ms	1.42 ± 0.48
stencil/dagger/N=256 (block 128)/multi-expr	3.9 ± 1.9 ms	2.49 ± 0.56 ms	1.57 ± 0.84
stencil/dagger/N=256 (block 128)/neighbors (Clamp)	2.37 ± 0.15 ms	2.66 ± 0.38 ms	0.889 ± 0.14
stencil/dagger/N=256 (block 128)/neighbors (Pad)	2.72 ± 0.92 ms	2.33 ± 0.15 ms	1.16 ± 0.4
stencil/dagger/N=256 (block 128)/neighbors (Reflect)	2.29 ± 0.1 ms	2.34 ± 0.3 ms	0.978 ± 0.13
stencil/dagger/N=256 (block 128)/neighbors (Wrap)	2.32 ± 0.16 ms	2.26 ± 0.18 ms	1.03 ± 0.11
stencil/dagger/N=256 (block 128)/update (+)	3.17 ± 2.1 ms	1.9 ± 0.59 ms	1.67 ± 1.2
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap)	1.96 ± 0.095 ms	2.17 ± 0.15 ms	0.902 ± 0.075
stencil/dagger/N=256 (block 256)/assign (const)	0.73 ± 0.11 ms	0.665 ± 0.079 ms	1.1 ± 0.22
stencil/dagger/N=256 (block 256)/multi-expr	1.29 ± 0.048 ms	1.83 ± 0.71 ms	0.707 ± 0.28
stencil/dagger/N=256 (block 256)/neighbors (Clamp)	1.81 ± 0.14 ms	1.77 ± 0.044 ms	1.03 ± 0.082
stencil/dagger/N=256 (block 256)/neighbors (Pad)	1.71 ± 0.086 ms	1.67 ± 0.087 ms	1.03 ± 0.075
stencil/dagger/N=256 (block 256)/neighbors (Reflect)	1.87 ± 0.17 ms	1.59 ± 0.19 ms	1.17 ± 0.18
stencil/dagger/N=256 (block 256)/neighbors (Wrap)	1.67 ± 0.13 ms	1.63 ± 0.11 ms	1.03 ± 0.1
stencil/dagger/N=256 (block 256)/update (+)	0.878 ± 0.15 ms	0.801 ± 0.12 ms	1.1 ± 0.25
time_to_load	1.04 ± 0.0059 s	1.06 ± 0.011 s	0.987 ± 0.012

Plots

⚠️ Regressions (> 25.0%)

linalg/dagger/N=256 (block 256)/cholesky: +82.3%
stencil/dagger/N=256 (block 256)/multi-expr: +41.4%
stencil/dagger/N=1024 (block 512)/neighbors (Wrap): +32.6%
linalg/dagger/N=256 (block 256)/syrk (A'*A): +28.2%
linalg/dagger/N=256 (block 128)/lu: +26.7%
array/dagger/N=256 (block 128)/reduce (sum): +25.2%

Improvements (> 25.0% faster)

stencil/dagger/N=256 (block 128)/update (+): -40.1%
stencil/dagger/N=256 (block 128)/multi-expr: -36.2%
array/dagger/N=256 (block 128)/add (X + X): -33.9%
stencil/dagger/N=256 (block 128)/assign (const): -29.6%

Distributed benchmarks (4 processes)

Dagger benchmarks: `dirty` vs `master`

	master	dirty	master / dirty
array/dagger/N=1024 (block 128)/add (X + X)	26.2 ± 0.0036 ms	26 ± 0.27 ms	1.01 ± 0.011
array/dagger/N=1024 (block 128)/alloc (rand)	14.2 ± 0.12 ms	14 ± 0.27 ms	1.02 ± 0.021
array/dagger/N=1024 (block 128)/broadcast (X .+ 1)	13.2 ± 0.029 ms	12.9 ± 0.049 ms	1.02 ± 0.0045
array/dagger/N=1024 (block 128)/map (sin.(X))	20.1 ± 0.2 ms	20.1 ± 0.019 ms	0.999 ± 0.0098
array/dagger/N=1024 (block 128)/norm	11.3 ± 0.33 ms	11.2 ± 0.49 ms	1.02 ± 0.053
array/dagger/N=1024 (block 128)/reduce (sum)	17.6 ± 0.016 ms	17.4 ± 0.0046 ms	1.01 ± 0.00096
array/dagger/N=1024 (block 128)/transpose (permutedims)	18.5 ± 0.12 ms	18.1 ± 0.072 ms	1.02 ± 0.008
array/dagger/N=1024 (block 512)/add (X + X)	6.08 ± 0.077 ms	6.16 ± 0.12 ms	0.988 ± 0.023
array/dagger/N=1024 (block 512)/alloc (rand)	5.11 ± 0.046 ms	5.04 ± 0.042 ms	1.01 ± 0.012
array/dagger/N=1024 (block 512)/broadcast (X .+ 1)	4.72 ± 0.14 ms	4.65 ± 0.075 ms	1.01 ± 0.034
array/dagger/N=1024 (block 512)/map (sin.(X))	11.8 ± 0.044 ms	11.9 ± 0.057 ms	0.991 ± 0.006
array/dagger/N=1024 (block 512)/norm	1.8 ± 0.055 ms	1.77 ± 0.024 ms	1.02 ± 0.034
array/dagger/N=1024 (block 512)/reduce (sum)	2.81 ± 0.043 ms	2.8 ± 0.014 ms	1 ± 0.016
array/dagger/N=1024 (block 512)/transpose (permutedims)	7.22 ± 0.18 ms	7.23 ± 0.073 ms	0.998 ± 0.027
array/dagger/N=256 (block 128)/add (X + X)	2.37 ± 0.042 ms	2.25 ± 0.06 ms	1.05 ± 0.034
array/dagger/N=256 (block 128)/alloc (rand)	1.93 ± 0.043 ms	1.82 ± 0.05 ms	1.06 ± 0.038
array/dagger/N=256 (block 128)/broadcast (X .+ 1)	1.14 ± 0.063 ms
array/dagger/N=256 (block 128)/map (sin.(X))	1.54 ± 0.02 ms	1.53 ± 0.079 ms	1.01 ± 0.054
array/dagger/N=256 (block 128)/norm	1.17 ± 0.2 ms	0.93 ± 0.028 ms	1.26 ± 0.22
array/dagger/N=256 (block 128)/reduce (sum)	2.11 ± 0.3 ms	1.68 ± 0.018 ms	1.26 ± 0.18
array/dagger/N=256 (block 128)/transpose (permutedims)	1.81 ± 0.036 ms	1.78 ± 0.0056 ms	1.02 ± 0.02
array/dagger/N=256 (block 256)/add (X + X)	1.43 ± 0.0051 ms	1.41 ± 0.0054 ms	1.02 ± 0.0053
array/dagger/N=256 (block 256)/alloc (rand)	1.32 ± 0.016 ms	1.31 ± 0.02 ms	1.01 ± 0.02
array/dagger/N=256 (block 256)/broadcast (X .+ 1)	0.678 ± 0.0039 ms	0.67 ± 0.027 ms	1.01 ± 0.042
array/dagger/N=256 (block 256)/map (sin.(X))	1.13 ± 0.017 ms	1.12 ± 0.024 ms	1.01 ± 0.026
array/dagger/N=256 (block 256)/norm	0.507 ± 0.018 ms	0.514 ± 0.0057 ms	0.986 ± 0.036
array/dagger/N=256 (block 256)/reduce (sum)	0.944 ± 0.025 ms	0.877 ± 0.049 ms	1.08 ± 0.067
array/dagger/N=256 (block 256)/transpose (permutedims)	1.14 ± 0.017 ms	1.16 ± 0.0092 ms	0.984 ± 0.017
linalg/dagger/N=1024 (block 128)/cholesky	0.0767 ± 0.0017 s	0.0769 ± 0.0018 s	0.997 ± 0.032
linalg/dagger/N=1024 (block 128)/lu	0.154 ± 0.0026 s	0.15 ± 0.0042 s	1.03 ± 0.034
linalg/dagger/N=1024 (block 128)/matmul (A*A)	0.137 ± 0.0013 s	0.137 ± 0.00045 s	1 ± 0.0099
linalg/dagger/N=1024 (block 128)/matvec (A*x)	23.2 ± 0.082 ms	22.8 ± 0.072 ms	1.02 ± 0.0048
linalg/dagger/N=1024 (block 128)/qr	0.238 ± 0.00081 s	0.234 ± 0.0007 s	1.02 ± 0.0046
linalg/dagger/N=1024 (block 128)/solve (A\b via lu)	0.195 ± 0.00035 s	0.19 ± 0.00068 s	1.03 ± 0.0041
linalg/dagger/N=1024 (block 128)/svd	56.3 s	55.4 s	1.01
linalg/dagger/N=1024 (block 128)/syrk (A'*A)	0.0965 ± 0.0013 s	0.0966 ± 0.0024 s	0.999 ± 0.029
linalg/dagger/N=1024 (block 512)/cholesky	19.3 ± 0.027 ms	18 ± 0.091 ms	1.07 ± 0.0056
linalg/dagger/N=1024 (block 512)/lu	0.0329 ± 0.00047 s	0.0344 ± 0.00033 s	0.958 ± 0.017
linalg/dagger/N=1024 (block 512)/matmul (A*A)	28.1 ± 3.8 ms	24.7 ± 0.26 ms	1.14 ± 0.15
linalg/dagger/N=1024 (block 512)/matvec (A*x)	3.87 ± 0.078 ms	3.76 ± 0.017 ms	1.03 ± 0.021
linalg/dagger/N=1024 (block 512)/qr	0.0914 ± 0.00017 s	0.0937 ± 0.00068 s	0.975 ± 0.0073
linalg/dagger/N=1024 (block 512)/solve (A\b via lu)	0.0396 ± 0.0012 s	0.037 ± 0.00066 s	1.07 ± 0.037
linalg/dagger/N=1024 (block 512)/svd	59.6 s	0.0168 h	0.988
linalg/dagger/N=1024 (block 512)/syrk (A'*A)	22.5 ± 0.048 ms	24.9 ± 3 ms	0.905 ± 0.11
linalg/dagger/N=256 (block 128)/cholesky	4.94 ± 0.042 ms	5.1 ± 0.03 ms	0.968 ± 0.01
linalg/dagger/N=256 (block 128)/lu	8.96 ± 0.16 ms	9.05 ± 0.00029 ms	0.991 ± 0.018
linalg/dagger/N=256 (block 128)/matmul (A*A)	6.23 ± 0.017 ms	6.07 ± 0.12 ms	1.03 ± 0.02
linalg/dagger/N=256 (block 128)/matvec (A*x)	2.92 ± 0.011 ms	2.91 ± 0.062 ms	1 ± 0.022
linalg/dagger/N=256 (block 128)/qr	8.18 ± 0.00045 ms	8.4 ± 0.46 ms	0.974 ± 0.053
linalg/dagger/N=256 (block 128)/solve (A\b via lu)	15.5 ± 0.22 ms	15.8 ± 0.28 ms	0.983 ± 0.022
linalg/dagger/N=256 (block 128)/svd	0.663 ± 0.0002 s	0.642 ± 0.013 s	1.03 ± 0.021
linalg/dagger/N=256 (block 128)/syrk (A'*A)	6.98 ± 0.16 ms	7.15 ± 0.038 ms	0.977 ± 0.023
linalg/dagger/N=256 (block 256)/cholesky	2.77 ± 0.019 ms	3.71 ± 0.072 ms	0.746 ± 0.015
linalg/dagger/N=256 (block 256)/lu	4.36 ± 0.015 ms	4.57 ± 0.058 ms	0.952 ± 0.012
linalg/dagger/N=256 (block 256)/matmul (A*A)	2.2 ± 0.0071 ms	2.4 ± 0.19 ms	0.915 ± 0.071
linalg/dagger/N=256 (block 256)/matvec (A*x)	1.47 ± 0.0033 ms	1.6 ± 0.014 ms	0.92 ± 0.0081
linalg/dagger/N=256 (block 256)/qr	4.95 ± 0.0028 ms	5.18 ± 0.073 ms	0.956 ± 0.013
linalg/dagger/N=256 (block 256)/solve (A\b via lu)	7.56 ± 0.013 ms	8.05 ± 0.0039 ms	0.939 ± 0.0017
linalg/dagger/N=256 (block 256)/svd	0.451 ± 0.0014 s	0.458 ± 0.00048 s	0.985 ± 0.0032
linalg/dagger/N=256 (block 256)/syrk (A'*A)	3.32 ± 0.051 ms	3.65 ± 0.041 ms	0.911 ± 0.017
stencil/dagger/N=1024 (block 128)/alloc (neighbors Wrap)	0.0424 ± 5.9e-05 s	0.0402 ± 0.00021 s	1.05 ± 0.0057
stencil/dagger/N=1024 (block 128)/assign (const)	10.3 ± 0.049 ms	10.2 ± 0.082 ms	1.01 ± 0.0094
stencil/dagger/N=1024 (block 128)/multi-expr	22.9 ± 0.55 ms	23.1 ± 0.23 ms	0.991 ± 0.026
stencil/dagger/N=1024 (block 128)/neighbors (Clamp)	31.5 ± 0.47 ms	31.5 ± 0.14 ms	1 ± 0.016
stencil/dagger/N=1024 (block 128)/neighbors (Pad)	31.2 ± 0.44 ms	0.0322 ± 0.00011 s	0.968 ± 0.014
stencil/dagger/N=1024 (block 128)/neighbors (Reflect)	30.9 ± 0.22 ms	0.0319 ± 1.9e-05 s	0.967 ± 0.0068
stencil/dagger/N=1024 (block 128)/neighbors (Wrap)	0.0321 ± 0.00017 s	0.0331 ± 0.00046 s	0.969 ± 0.014
stencil/dagger/N=1024 (block 128)/update (+)	13.2 ± 0.13 ms	13.5 ± 0.12 ms	0.975 ± 0.013
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap)	13.9 ± 0.046 ms	12 ± 0.14 ms	1.15 ± 0.014
stencil/dagger/N=1024 (block 512)/assign (const)	2.35 ± 0.041 ms	2.39 ± 0.052 ms	0.982 ± 0.027
stencil/dagger/N=1024 (block 512)/multi-expr	5 ± 0.042 ms	5.01 ± 0.061 ms	0.997 ± 0.015
stencil/dagger/N=1024 (block 512)/neighbors (Clamp)	11 ± 0.068 ms	11.4 ± 0.14 ms	0.965 ± 0.013
stencil/dagger/N=1024 (block 512)/neighbors (Pad)	11.9 ± 1.2 ms	12.6 ± 0.068 ms	0.944 ± 0.097
stencil/dagger/N=1024 (block 512)/neighbors (Reflect)	10.8 ± 0.011 ms	11.6 ± 0.11 ms	0.934 ± 0.0086
stencil/dagger/N=1024 (block 512)/neighbors (Wrap)	12.2 ± 0.065 ms	13.1 ± 0.037 ms	0.934 ± 0.0056
stencil/dagger/N=1024 (block 512)/update (+)	3.08 ± 0.04 ms	3.14 ± 0.023 ms	0.982 ± 0.015
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap)	3.32 ± 0.0069 ms	3.16 ± 0.014 ms	1.05 ± 0.005
stencil/dagger/N=256 (block 128)/assign (const)	1.27 ± 0.012 ms	1.23 ± 0.0028 ms	1.03 ± 0.01
stencil/dagger/N=256 (block 128)/multi-expr	2.51 ± 0.082 ms	2.5 ± 0.04 ms	1 ± 0.036
stencil/dagger/N=256 (block 128)/neighbors (Clamp)	2.64 ± 0.038 ms	2.72 ± 0.041 ms	0.972 ± 0.02
stencil/dagger/N=256 (block 128)/neighbors (Pad)	2.57 ± 0.0003 ms	2.7 ± 0.0051 ms	0.951 ± 0.0018
stencil/dagger/N=256 (block 128)/neighbors (Reflect)	2.64 ± 0.037 ms	2.71 ± 0.024 ms	0.976 ± 0.016
stencil/dagger/N=256 (block 128)/neighbors (Wrap)	2.78 ± 0.12 ms	2.76 ± 0.052 ms	1.01 ± 0.048
stencil/dagger/N=256 (block 128)/update (+)	1.5 ± 0.0068 ms	1.57 ± 0.072 ms	0.951 ± 0.044
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap)	1.97 ± 0.0025 ms	1.87 ± 0.026 ms	1.05 ± 0.015
stencil/dagger/N=256 (block 256)/assign (const)	0.814 ± 0.013 ms	0.815 ± 0.014 ms	0.998 ± 0.024
stencil/dagger/N=256 (block 256)/multi-expr	1.6 ± 0.044 ms	1.6 ± 0.0031 ms	0.998 ± 0.028
stencil/dagger/N=256 (block 256)/neighbors (Clamp)	1.69 ± 0.018 ms	1.79 ± 0.016 ms	0.94 ± 0.013
stencil/dagger/N=256 (block 256)/neighbors (Pad)	1.67 ± 0.0099 ms	1.81 ± 0.021 ms	0.924 ± 0.012
stencil/dagger/N=256 (block 256)/neighbors (Reflect)	1.69 ± 0.03 ms	1.73 ± 0.00088 ms	0.979 ± 0.018
stencil/dagger/N=256 (block 256)/neighbors (Wrap)	1.83 ± 0.072 ms	1.84 ± 0.015 ms	0.993 ± 0.04
stencil/dagger/N=256 (block 256)/update (+)	1.05 ± 0.0046 ms	1.08 ± 0.0079 ms	0.973 ± 0.0083
time_to_load	1.18 ± 0.0014 s	1.17 ± 0.0051 s	1.01 ± 0.0046

No regressions beyond 35.0% 🎉

MPI benchmarks (4 ranks)

Dagger benchmarks: `dirty` vs `master`

	master	dirty	master / dirty
array/dagger/N=1024 (block 128)/add (X + X)	0.58 ± 0.0012 s
array/dagger/N=1024 (block 128)/alloc (rand)	0.0931 ± 0.014 s
array/dagger/N=1024 (block 128)/broadcast (X .+ 1)	0.0423 ± 0.00052 s
array/dagger/N=1024 (block 128)/map (sin.(X))	0.0432 ± 0.00084 s
array/dagger/N=1024 (block 128)/norm	0.0447 ± 0.00049 s
array/dagger/N=1024 (block 128)/reduce (sum)	0.107 ± 0.0058 s
array/dagger/N=1024 (block 128)/transpose (permutedims)	0.273 ± 0.012 s
array/dagger/N=1024 (block 512)/add (X + X)	0.0429 ± 0.0024 s
array/dagger/N=1024 (block 512)/alloc (rand)	0.0512 ± 0.0021 s
array/dagger/N=1024 (block 512)/broadcast (X .+ 1)	4.23 ± 0.53 ms
array/dagger/N=1024 (block 512)/map (sin.(X))	7.45 ± 0.54 ms
array/dagger/N=1024 (block 512)/norm	5.6 ± 0.035 ms
array/dagger/N=1024 (block 512)/reduce (sum)	7.51 ± 0.0055 ms
array/dagger/N=1024 (block 512)/transpose (permutedims)	23.9 ± 0.031 ms
array/dagger/N=256 (block 128)/add (X + X)	0.0394 ± 0.00041 s
array/dagger/N=256 (block 128)/alloc (rand)	22 ± 15 ms
array/dagger/N=256 (block 128)/broadcast (X .+ 1)	3.34 ± 0.13 ms
array/dagger/N=256 (block 128)/map (sin.(X))	3.47 ± 0.02 ms
array/dagger/N=256 (block 128)/norm	3.63 ± 0.072 ms
array/dagger/N=256 (block 128)/reduce (sum)	6.93 ± 0.038 ms
array/dagger/N=256 (block 128)/transpose (permutedims)	20.1 ± 2.2 ms
array/dagger/N=256 (block 256)/add (X + X)	11 ± 0.13 ms
array/dagger/N=256 (block 256)/alloc (rand)	2.81 ± 0.095 ms
array/dagger/N=256 (block 256)/broadcast (X .+ 1)	1.3 ± 0.056 ms
array/dagger/N=256 (block 256)/map (sin.(X))	2.06 ± 0.026 ms
array/dagger/N=256 (block 256)/norm	1.87 ± 0.012 ms
array/dagger/N=256 (block 256)/reduce (sum)	2.17 ± 0.038 ms
array/dagger/N=256 (block 256)/transpose (permutedims)	5.47 ± 0.00064 ms
linalg/dagger/N=1024 (block 128)/cholesky	15.6 s
linalg/dagger/N=1024 (block 128)/lu	0.139 h
linalg/dagger/N=1024 (block 128)/matmul (A*A)	0.0175 h
linalg/dagger/N=1024 (block 128)/matvec (A*x)	0.868 ± 0.024 s
linalg/dagger/N=1024 (block 128)/qr	27.7 s
linalg/dagger/N=1024 (block 128)/solve (A\b via lu)	0.138 h
linalg/dagger/N=1024 (block 128)/syrk (A'*A)	36.8 s
linalg/dagger/N=1024 (block 512)/cholesky	0.51 s
time_to_load	1.11 ± 0.011 s	1.11 ± 0.032 s	0.994 ± 0.03

No regressions beyond 35.0% 🎉

Full results and plots (download the benchmark-results-* artifacts).

…ronment MPI.mpiexec(identity) only extracted the executable/Cmd; MPI.jl's mpiexec() sets up the environment the binary needs (library paths for the bundled MPICH_jll, etc.) only for the dynamic extent of its callback, so running the extracted command afterwards spawned a process missing that environment, which died immediately before worker_mpi.jl could produce any results. Move the run(...; wait=false) call inside the `do` block, mirroring test/run_mpi.jl.

Sweep BENCHMARK_BLOCKSIZE instead of pinning one tile. The three benchmark jobs ran BENCHMARK_SCALE='[256, 1024]' with the tile left at square_block's default of 512, which makes N=256 a *single* chunk (it returns N when N <= tile) and N=1024 only 2x2 -- so anything touching data distribution, halo exchange or cross-space transfer was a no-op at one size and barely exercised at the other. Simply lowering the tile would trade one blind spot for another: tile size sets the ratio of per-tile task overhead to bytes moved per transfer, and a data-movement change can be a large win at one tile and a regression at the other. Measured under MPI on 4 ranks, commit c60e294 is ~25% slower at tile 128 and ~1.5-2x faster at tile 512 -- opposite conclusions from the same code. So BENCHMARK_BLOCKSIZE now takes a list, like BENCHMARK_SCALE. common.jl gains `blocksizes` and `blocks_for(N)`, and the three dense suites gain an inner loop over it. `blocks_for` de-duplicates tiles that collapse to the same block at a given N (at N=256 both 512 and 256 give 256), which matters because the groups are keyed "N=$N (block $b)" and would otherwise collide. CI uses '[128, 512]'; an unset variable still means a single tile of 512, as before.

`gather_free_syncdeps!` decided whether it could inspect a free candidate's backing buffer via `aliasing_available`, which under MPI was true only on the rank owning the ref. But `aliasing` is *collective* there -- the owner computes and broadcasts -- so the owning rank alone would enter a broadcast no other rank was in. That consumed a tag on one rank, desynchronizing every subsequent `to_tag`, and then deadlocked in the copy cycles at region end. Symptom on this branch: 26 errors per rank in `test/mpi.jl` (Cholesky, LU, and every `@stencil` case), all `ArgumentError: Non-uniform value` raised from `take_ref_id!`. Bisected to 87273af -- not because that commit is wrong, but because ranking by written args moves each task onto the rank owning the chunk it writes, which is what first makes a buffer underlie only wrapper arguments and so reach this branch at all. Every rank now takes the rank-uniform `key_ainfo` fallback whenever execution is uniform, so the decision depends solely on `uniform_execution()`. That leaves `aliasing_available` with no remaining call site -- `uniform_execution(::MPIAcceleration)` is unconditionally `true`, so the MPIRef method could never be reached past the short-circuit -- so it is removed along with its generic fallback. `test/mpi.jl` goes from 26 errors per rank to 0 (395/397/385/373 passing, no failures); Distributed datadeps is unchanged at 1272 passing. Also adds a "Partial slot currency" testset covering the case that motivated this: writing through a `ChunkView` on a non-owner rank and then reading the whole parent there, which must see the view's write and pull the rest from the owner. Its expectation is snapshotted *before* the region -- `A` is a non-Chunk argument, so rank 0 owns it and Datadeps writes back into it in place; deriving the baseline afterwards folds the update in and then adds it again (and only rank 0's copy is written back, so it would also disagree across ranks).

`gather_free_syncdeps!`'s uniform-execution fallback syncs on the cache key ainfo, but guarded that on the key being tracked: haskey(state.ainfos_overlaps, wrapped) && get_write_deps!(state, space, wrapped, write_num, syncdeps) The key ainfo describes the original object in its *source* space, so it is frequently not one of the tracked destination-space ainfos -- rank-stamped spans mean it cannot overlap them, and the interval tree cannot be queried with it either. When that happens the guard short-circuits and the `unsafe_free!` task is spawned with an **empty syncdep set**, racing every task still reading the buffer. Instrumenting a 64x64 / Blocks(16,16) Cholesky under 4 ranks shows a handful of such frees per rank (`nsyncdeps=0`) among ~100 with 2-3. This is normally invisible: the freed Libc block is recycled by the next `alloc_libc_array`, and `move_rewrap` immediately overwrites it with a full copy of the real data, so a use-after-free read gets plausible -- finite, usually identical -- values. The untracked case now syncs against every tracked ainfo in the space instead: a strict over-approximation of what can overlap the buffer, and cheap, since frees happen once per buffer at region end and only untracked keys take this branch. `get_write_deps!` already covers both writers and readers, so the tracked case was fine as-is. `test/mpi.jl` is unchanged at 395/397/385/373 passing, no failures or errors.

A Datadeps slot's contents are never assumed to be current. `generate_slot!` says so outright -- it deliberately does not sync with the owner -- and `compute_remainder_for_arg!` decides what to copy purely from `arg_history` / `arg_owner`, never from the buffer. A slot in a space that does not yet appear in the argument's history is therefore always filled by a copy-to (`FullCopy`, or a span-exact `MultiRemainderAliasing` once part of it is current) before any task can read it. So for a dense, isbits payload crossing a space boundary, `move_rewrap` was sending bytes that were guaranteed to be overwritten. It now allocates the destination buffer and sends nothing: only the element type and dimensions travel (broadcast like a header under MPI, closure arguments under Distributed). The allocation is Libc-backed straight from `alloc_libc_array` rather than allocated and then copied into Libc memory by `libc_backed`, since there is nothing to copy. Under MPI that also fixes a latent no-op -- `libc_backed` was being applied to a `Chunk`, which hits the identity method. `from_space != to_space` is the load-bearing guard. When the two coincide the leaf transfer is the identity, so the "slot" *is* the original data and `compute_remainder_for_arg!` returns `NoAliasing()`; handing back a fresh buffer there would silently discard the argument. Allocation also stays routed through `aliased_object!`, so an already-present parent is still reused rather than shadowed by a detached buffer -- which is what keeps an array and its view sharing one destination allocation. N.B. This depends on the preceding free-syncdeps fix. Transferring the slot was masking a use-after-free: a recycled Libc block was immediately overwritten with real data, so a racing read returned plausible values. Without the transfer the same read hits uninitialized memory, which is how that bug first became visible (NaN out of a distributed Cholesky). Measured on 4 MPI ranks, a 1024x1024 `@stencil` sweep with `@neighbors`: 689ms/sweep at 128x128 tiles (872ms before) and 316ms at 512x512 tiles (509ms before). `test/mpi.jl` 398/403/388/376 passing; Distributed datadeps 1272 passing. Test contract updated accordingly: cross-space slots are no longer asserted to hold a copy of the source, only to have the right shape, type, and parent sharing; same-space slots must still be the real data. Two new testsets cover the allocation itself and the partial-currency invariant it rests on -- a task reading spans never written in its own space.

The untracked-key fallback added in the previous commit scans every tracked ainfo, and the free loop ran it once per buffer. A region with many untracked keys therefore paid O(frees x ainfos x overlap-set) at region end -- fine for the small regions in the test suite, but enough to turn a large MPI region into a timeout on a slow or oversubscribed runner. The scan does not depend on the buffer, only on the space, so it is hoisted into `all_free_syncdeps_for_space` and memoized by the caller for the duration of the free loop. Both call sites (flat `distribute_tasks!` and hierarchical `_hierarchical_copy_from_and_free!`) pass a per-loop `Dict`; the parameter is optional and defaults to no memoization, so the function is still correct when called without one. Behaviour is unchanged -- the same conservative set, computed once instead of repeatedly. `test/mpi.jl` 398/403/388/376 passing, no failures or errors, and the suite's wall clock is unchanged (12m07s vs 12m04s).

`mpi_endpoint_alloc` sent the destination buffer's dimensions with `bcast_yield`, treating them like header metadata. That made every rank a participant in a transfer that concerns only two, and `bcast_yield` is only safe at sequential, non-overlapping points: an intermediate rank cannot forward the tree while it is blocked inside a task awaiting dispatch, which closes the cross-rank wait cycle documented above `bcast_tree_children` -- the hang previously seen on Julia 1.10/1.11, and reported again on the 4-rank CI job. Only the destination rank needs the dimensions, so send them exactly the way the `mpi_endpoint_transfer` this replaces sent its payload: one point-to-point message between the same two ranks, just header-sized instead of the whole array. Every rank still takes a tag first, so the sequence stays rank-uniform. The same-rank case (e.g. CPU -> GPU within one rank) needs no message at all. On Julia 1.10 this also fixes a real miscompare, not just the hang: with the broadcast, `test/mpi.jl`'s "ChunkView dataflow" lost the `add1!` written through `cv_top` (rows short by exactly 1) on every rank. 1.12 did not show it. Julia 1.10, 4 ranks: 396+2 failures -> 398/403/388/376, no failures Julia 1.12, 4 ranks: 398/403/388/376, no failures (unchanged)

test/mpi.jl at 4 ranks passes cleanly (398/398) in ~12 minutes on an idle workstation, matching the ~12m benchmark in c13465a's commit message -- no reproducible deadlock. CI reportedly runs ~40 minutes and occasionally exceeds the 60-minute timeout, consistent with slower/shared runners eating the margin rather than a hang. Bump 4-rank's budget to 90 minutes; 2-rank stays at 60 since it isn't reported as tight.

move_rewrap runs from generate_slot! inside the per-task planning loop, and each task is handed to eager_launch! (and so starts executing, concurrently) before the loop moves on to plan the next task's header broadcast. That means this broadcast is not at the "sequential, non-overlapping point" the raw bcast_yield tree assumes -- it can race an already-dispatched task's own execute!/poolget activity, closing the same forwarder wait-cycle documented above bcast_meta_yield. Confirmed: a 4-rank mpiexec run of test/mpi.jl hung almost immediately with the old bcast_yield header broadcast under load from a concurrent MPI job on the same machine, and passed cleanly (16m44s, 398/398, no warnings) after switching to bcast_meta_yield. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The 4-rank CPU MPI job has been running out its CI timeout on Julia 1.10 and 1.11 with nothing in the log to point at. It reproduces once the run is given CI's CPU budget rather than a workstation's: pinned to 4 CPUs (what ubuntu-latest provides), Julia 1.10 at HEAD deadlocks outright, and SIGUSR1 backtrace dumps catch all four ranks inside `MPI_Barrier`, called from `check_uniform`. Nothing here is version-specific -- it is a race whose odds turn on GC and scheduling timing, which is why 1.12 slips through. Three defects, in decreasing order of severity: * `check_uniform` ended with `MPI.Barrier`. That is a blocking `ccall`, so it parks the OS thread inside MPI: the thread runs no other Julia task -- notably not `bcast_relay_loop`, which other ranks depend on to have a broadcast forwarded -- and it never reaches a GC safepoint, so any other thread requesting a collection stalls the process until the barrier returns. Rank r parks here waiting for rank s while s waits on a broadcast only r's relay can forward, and neither can move. The barrier bought nothing either: `compare_all` directly above it is already an arrival barrier, since it cannot return until every rank has entered it and sent. * `bcast_slot_wait` kept a one-shot slot keyed by tag alone. A tag does not identify a single broadcast -- `to_tag()` returns the planning task's thunk id, so every `move_rewrap` header broadcast issued while generating that task's slots shares it, one per wrapper level and one per argument. The second delivery overwrote the first before its consumer read it; that consumer then took the wrong value and the next one waited forever on a payload that had already arrived. Now a FIFO keyed by (root, tag), drained in the rank-uniform order MPI's non-overtaking guarantee establishes. * The relay busy-spun on `MPI_Improbe`, taking 107 of one thread's 127 profile samples -- a core per rank, which a 4-vCPU runner hosting 4 ranks cannot spare. It now spins hot through a burst and backs off after it, keeping the added latency off all but the first message of the next burst. Both waits that had no deadlock detection now have it, so a future regression here reports itself instead of silently spending the CI budget. Verified on Julia 1.10, 4 ranks x 2 threads pinned to 4 CPUs. HEAD deadlocks in the Stencils testset after ~35 minutes: ranks 1-3 time out on a `compare_all` recv from rank 0 on tag 1073741823 (`MPI.tag_ub()`) while rank 0, sitting in the undetected barrier, reports nothing at all, and `mpiexec` never exits. With this change the suite passes on all four ranks twice over, in 18m46s and 18m19s, with the expected 398/403/376/388 test counts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

jpsamaroo added 2 commits August 13, 2026 13:54

datadeps/hierarchical: Rank by written args separately during partiti…

87273af

…on_dag

DArray/stencil: Avoid xfer of full chunks for boundary across differe…

c60e294

…nt spaces

jpsamaroo added performance stencils labels Aug 13, 2026

CI/benchmarks: Add Distributed and MPI benchmarks

3c0b231

jpsamaroo force-pushed the jps/mpi-bad-scale-stencil branch from 3e7eedb to 3c0b231 Compare August 17, 2026 22:36

jpsamaroo and others added 10 commits August 17, 2026 15:56

Navigation Menu

Uh oh!

Uh oh!

Fix stencils over-transfer of neighbor data - #737

Fix stencils over-transfer of neighbor data#737
jpsamaroo wants to merge 13 commits into
masterfrom
jps/mpi-bad-scale-stencil

jpsamaroo commented Aug 13, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026 •

edited

Loading

Dagger benchmarks: `dirty` vs `master`

Plots

⚠️ Regressions (> 25.0%)

Improvements (> 25.0% faster)

Dagger benchmarks: `dirty` vs `master`

Dagger benchmarks: `dirty` vs `master`

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

jpsamaroo commented Aug 13, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dagger benchmarks: dirty vs master

Dagger benchmarks: dirty vs master

Plots

⚠️ Regressions (> 25.0%)

Improvements (> 25.0% faster)

Dagger benchmarks: dirty vs master

Dagger benchmarks: dirty vs master

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Aug 13, 2026 •

edited

Loading

Dagger benchmarks: `dirty` vs `master`

Dagger benchmarks: `dirty` vs `master`

Dagger benchmarks: `dirty` vs `master`

Dagger benchmarks: `dirty` vs `master`