Useful or not, from you.
pySCENIC distributed.core - ERROR
/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/arboreto/algo.py:214: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  expression_matrix = expression_data.as_matrix()
creating dask graph
48 partitions
computing dask graph
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/protocol/core.py", line 108, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/protocol/core.py", line 197, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_raw_false)
  File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
ValueError: 102826 exceeds max_map_len(32768)
distributed.core - ERROR - 102826 exceeds max_map_len(32768)
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/core.py", line 386, in handle_stream
    msgs = yield comm.read()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 206, in read
    deserializers=deserializers)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/comm/utils.py", line 79, in from_frames
    res = _from_frames()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/comm/utils.py", line 65, in _from_frames
    deserializers=deserializers)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/protocol/core.py", line 108, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/protocol/core.py", line 197, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_raw_false)
  File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
ValueError: 102826 exceeds max_map_len(32768)
shutting down client and local cluster
distributed.core - ERROR - 102826 exceeds max_map_len(32768)
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/core.py", line 346, in handle_comm
    result = yield result
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/scheduler.py", line 2018, in add_client
    yield self.handle_stream(comm=comm, extra={'client': client})
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/core.py", line 386, in handle_stream
    msgs = yield comm.read()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 206, in read
    deserializers=deserializers)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/comm/utils.py", line 79, in from_frames
    res = _from_frames()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/comm/utils.py", line 65, in _from_frames
    deserializers=deserializers)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/protocol/core.py", line 108, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/protocol/core.py", line 197, in loads_msgpack
    return msgpack.loads(payload, use_list=False, **msgpack_raw_false)
  File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
ValueError: 102826 exceeds max_map_len(32768)
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 970, in error_callback
    future.result()
concurrent.futures._base.CancelledError
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:33045 remote=tcp://127.0.0.1:58286>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:33435 remote=tcp://127.0.0.1:58286>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:33481 remote=tcp://127.0.0.1:58286>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:33482 remote=tcp://127.0.0.1:58286>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:33483 remote=tcp://127.0.0.1:58286>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:33484 remote=tcp://127.0.0.1:58286>
distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:33485 remote=tcp://127.0.0.1:58286>
finished
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/client.py", line 1487, in _gather
    st = self.futures[key]
KeyError: 'finalize-7b1845663f7c382673df6fc49437374f'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script.py", line 44, in <module>
    adjacencies = grnboost2(ex_matrix, tf_names=tf_names, verbose=True)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/arboreto/algo.py", line 41, in grnboost2
    early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/arboreto/algo.py", line 135, in diy
    .compute(graph, sync=True) \
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/client.py", line 2492, in compute
    result = self.gather(futures)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/client.py", line 1652, in gather
    asynchronous=asynchronous)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/client.py", line 670, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/utils.py", line 277, in sync
    six.reraise(*error[0])
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/utils.py", line 262, in f
    result[0] = yield future
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/distributed/client.py", line 1493, in _gather
    None)
  File "/home/user/anaconda3/envs/pyscenic_env/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
concurrent.futures._base.CancelledError: finalize-7b1845663f7c382673df6fc49437374f

How could I resolve this issue? Any suggestion helps.. Thank you very much. EL

That's a useful answer
Without any help

Dear Hicham,

I tried to reproduce the problem using a fresh installation of pyscenic (version 0.8.16) in a virgin miniconda environment (on a Linux RedHat distribution running on a dual Intel Xeon E5-2680 v3 machine). I installed the latest version of tornado (5.1.1) and ran grnboost from the command line. I failed to get this error message.

This is my list of packages installed:

arboreto==0.1.5
attrs==18.2.0
boltons==18.0.1
certifi==2018.10.15
Click==7.0
cloudpickle==0.6.1
cycler==0.10.0
cytoolz==0.9.0.1
dask==0.20.2
decorator==4.3.0
dill==0.2.8.2
distributed==1.24.2
frozendict==1.2
h5py==2.8.0
HeapDict==1.0.0
interlap==0.2.6
kiwisolver==1.0.1
llvmlite==0.25.0
loompy==2.0.2
matplotlib==3.0.2
msgpack==0.5.6
multiprocessing-on-dill==3.5.0a4
networkx==2.2
numba==0.40.1
numpy==1.15.4
pandas==0.23.4
psutil==5.4.8
pyarrow==0.11.1
pyparsing==2.3.0
pyscenic==0.8.16
python-dateutil==2.7.5
pytz==2018.7
PyYAML==3.13
scikit-learn==0.20.1
scipy==1.1.0
six==1.11.0
sortedcontainers==2.1.0
tblib==1.3.2
toolz==0.9.0
tornado==5.1.1
tqdm==4.28.1
typing==3.6.6
umap-learn==0.3.6
zict==0.1.3

Anyhow, this problem is related to the GRNBoost2 step of pySCENIC which is provided through the arboreto package. This issue is already registered there: tmoerman/arboreto#12 .

Kindest regards, Bram

Hi Bram, I followed the instruction in https://pyscenic.readthedocs.io/en/latest/#run-genie3-or-grnboost-from-arboreto-to-infer-co-expression-modules, and ended up with the error "concurrent.futures._base.CancelledError" mentioned above when running the grnboost2 function. I'm quite new to tornado. And what I want to know is that is there any alternative function or certain workaround to avoid this error? Thanks! Best regards, Yingyong