Conflict with for-loop and explicit return in pmap

[JamesWrigley]

I've run into a strange edge case with @showprogress and pmap. First I import packages and add a worker

julia> using ProgressMeter
julia> using Distributed
julia> addprocs(1)                                                                                                                                                                                                                                                                                                             
1-element Vector{Int64}:                                                                                                                                                                                                                                                                                                       
 2

Then I try running this:

julia> @showprogress for i = 1:10                                                                                                                                                                                                                                                                                                                                                                                                         
           pmap(1:10) do j                                                                                                                                                                                                                                                                                                                                                                                                                
               return j                                                                                                                                                                                                                                                                                                                                                                                                                   
           end                                                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                          
           sleep(0.1)                                                                                                                                                                                                                                                                                                                                                                                                                     
       end

Which fails with this very long error about not being able to find ProgressMeter:

First error message

julia> @showprogress for i = 1:10                                                                                                                                                                                                                                                                                                                                                                                                         
           pmap(1:10) do j                                                                                                                                                                                                                                                                                                                                                                                                                
               return j                                                                                                                                                                                                                                                                                                                                                                                                                   
           end                                                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                          
           sleep(0.1)                                                                                                                                                                                                                                                                                                                                                                                                                     
       end
ERROR: On worker 2:                                                                                                                                                                                                                                                                                                                                                                                                                       
KeyError: key ProgressMeter [92933f4c-e287-5a05-a399-4b506db050ca] not found                                                                                                                                                                                                                                                                                                                                                              
Stacktrace:                                                                                                                                                                                                                                                                                                                                                                                                                               
  [1] getindex                                                                                                                                                                                                                                                                                                                                                                                                                            
    @ ./dict.jl:498 [inlined]                                                                                                                                                                                                                                                                                                                                                                                                             
  [2] macro expansion                                                                                                                                                                                                                                                                                                                                                                                                                     
    @ ./lock.jl:223 [inlined]                                                                                                                                                                                                                                                                                                                                                                                                             
  [3] root_module                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ./loading.jl:1254                                                                                                                                                                                                                                                                                                                                                                                                                   
  [4] deserialize_module                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:993                                                                                                                                                                                                                                                                                                                               
  [5] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:895                                                                                                                                                                                                                                                                                                                               
  [6] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813                                                                                                                                                                                                                                                                                                                               
  [7] deserialize_datatype                                                                                                                                                                                                                                                                                                                                                                                                                
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1363                                                                                                                                                                                                                                                                                                                              
  [8] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:866                                                                                                                                                                                                                                                                                                                               
  [9] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813                                                                                                                                                                                                                                                                                                                               
 [10] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:873                                                                                                                                                                                                                                                                                                                               
 [11] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813                                                                                                                                                                                                                                                                                                                               
 [12] deserialize_expr                                                                                                                                                                                                                                                                                                                                                                                                                    
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1266                                                                                                                                                                                                                                                                                                                              
 [13] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:893                                                                                                                                                                                                                                                                                                                               
 [14] deserialize_fillarray!                                                                                                                                                                                                                                                                                                                                                                                                              
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1256                                                                                                                                                                                                                                                                                                                              
 [15] deserialize_array                                                                                                                                                                                                                                                                                                                                                                                                                   
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1248                                                                                                                                                                                                                                                                                                                              
 [16] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:864                                                                                                                                                                                                                                                                                                                               
 [17] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]                                                                                                                                                                                                                                                                                                                     
 [18] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1129                                                                                                                                                                                                                                                                                                                              
 [19] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:959                                                                                                                                                                                                                                                                                                                               
 [20] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]                                                                                                                                                                                                                                                                                                                     
 [21] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1040                                                                                                                                                                                                                                                                                                                              
 [22] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:959                                                                                                                                                                                                                                                                                                                               
 [23] deserialize_fillarray!                                                                                                                                                                                                                                                                                                                                                                                                              
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1256                                                                                                                                                                                                                                                                                                                              
 [24] deserialize_array                                                                                                                                                                                                                                                                                                                                                                                                                   
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1248                                                                                                                                                                                                                                                                                                                              
 [25] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:864                                                                                                                                                                                                                                                                                                                               
 [26] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]                                                                                                                                                                                                                                                                                                                     
 [27] deserialize_typename                                                                                                                                                                                                                                                                                                                                                                                                                
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1326                                                                                                                                                                                                                                                                                                                              
 [28] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/clusterserialize.jl:68                                                                                                                                                                                                                                                                                                                               
 [29] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:959                                                                                                                                                                                                                                                                                                                               
 [30] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813                                                                                                                                                                                                                                                                                                                               
 [31] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:870                                                                                                                                                                                                                                                                                                                               
 [32] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813                                                                                                                                                                                                                                                                                                                               
 [33] handle_deserialize                                                                                                                                                                                                                                                                                                                                                                                                                  
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:873                                                                                                                                                                                                                                                                                                                               
 [34] deserialize                                                                                                                                                                                                                                                                                                                                                                                                                         
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]                                                                                                                                                                                                                                                                                                                     
 [35] deserialize_msg                                                                                                                                                                                                                                                                                                                                                                                                                     
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/messages.jl:87                                                                                                                                                                                                                                                                                                                                       
 [36] #invokelatest#2                                                                                                                                                                                                                                                                                                                                                                                                                     
    @ ./essentials.jl:729 [inlined]                                                                                                                                                                                                                                                                                                                                                                                                       
 [37] invokelatest                                                                                        
    @ ./essentials.jl:726 [inlined]                                                                       
 [38] message_handler_loop                                                                                
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:176                                                                                                         
 [39] process_tcp_streams                                                                                 
    @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133                                                                                                         
 [40] #103                                                                                                
    @ ./task.jl:484                                                                                       
Stacktrace:                                                                                               
  [1] (::Base.var"#938#940")(x::Task)                                                                     
    @ Base ./asyncmap.jl:177                                                                              
  [2] foreach(f::Base.var"#938#940", itr::Vector{Any})                                                    
    @ Base ./abstractarray.jl:2774                                                                        
  [3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::UnitRange{Int64})                                                                                                                                                        
    @ Base ./asyncmap.jl:177                                                                                                   
  [4] wrap_n_exec_twice                                                                                                        
    @ ./asyncmap.jl:153 [inlined]                                                                                              
  [5] #async_usemap#923                                                                                                        
    @ ./asyncmap.jl:103 [inlined]                                                                                              
  [6] #asyncmap#922                                                                                                            
    @ ./asyncmap.jl:81 [inlined]                                                                                               
  [7] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{Any}, retry_check::Nothing)                                                                                                                                                                                                                                                                                                                                                          
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:126                                                                                                                                                   
  [8] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64})                                                                    
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:99                                                                                                                                                    
  [9] pmap(f::Function, c::UnitRange{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})                                                                                                                                            
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156                                                                                                                                                   
 [10] pmap(f::Function, c::UnitRange{Int64})                                                                                   
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156                                                                                                                                                   
 [11] macro expansion                                                                                                          
    @ REPL[10]:2 [inlined]                                                                                                     
 [12] top-level scope                                                                                                          
    @ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:938

Then I import ProgressMeter everywhere and try again:

julia> @everywhere using ProgressMeter
julia> @showprogress for i = 1:10
           pmap(1:10) do j
               return j
           end
           
           sleep(0.1)
       end

This fails with a different error:

Second error message

julia> @showprogress for i = 1:10
           pmap(1:10) do j
               return j
           end
           
           sleep(0.1)
       end
ERROR: On worker 2:
UndefVarError: ##meter#313 not defined
Stacktrace:
 [1] macro expansion
   @ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:710 [inlined]
 [2] #3
   @ ./REPL[17]:3
 [3] #invokelatest#2
   @ ./essentials.jl:729
 [4] invokelatest
   @ ./essentials.jl:726
 [5] #110
   @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
 [6] run_work_thunk
   @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
 [7] macro expansion
   @ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
 [8] #109
   @ ./task.jl:484
Stacktrace:
  [1] (::Base.var"#938#940")(x::Task)
    @ Base ./asyncmap.jl:177
  [2] foreach(f::Base.var"#938#940", itr::Vector{Any})
    @ Base ./abstractarray.jl:2774
  [3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::UnitRange{Int64})
    @ Base ./asyncmap.jl:177
  [4] wrap_n_exec_twice
    @ ./asyncmap.jl:153 [inlined]
  [5] #async_usemap#923
    @ ./asyncmap.jl:103 [inlined]
  [6] #asyncmap#922
    @ ./asyncmap.jl:81 [inlined]
  [7] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{Any}, retry_check::Nothing)
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:126
  [8] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64})
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:99
  [9] pmap(f::Function, c::UnitRange{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156
 [10] pmap(f::Function, c::UnitRange{Int64})
    @ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156
 [11] macro expansion
    @ REPL[17]:2 [inlined]
 [12] top-level scope
    @ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:938

But this works fine, even without @everywhere using ProgressMeter:

julia> @showprogress for i = 1:10
           pmap(1:10) do j
               j        # There is no explicit return here
           end
           
           sleep(0.1)
       end

For some reason the omission of return in the pmap body makes it work. I had a quick look through the source to see if I could find the problem, and I think it's something to do with showprogress_process_expr(): https://github.com/timholy/ProgressMeter.jl/blob/v1.7.2/src/ProgressMeter.jl#L707

As far as I can understand it, the code is recursively going through the loop body, and in this case where it finds a break/return it inserts a special node to handle break and return statements. The problem is that when there's a return in some code that's technically part of the loop but isn't actually executed in the loop (e.g. a closure), the macro will still insert those special nodes. I guess that explains the second error, UndefVarError: ##meter#313 not defined; the meter#313 variable is probably metersym in the code, which was created on the master process and then inserted into the AST to be executed on the worker.

I saw that #146 has similar error messages, but I don't think this is related.