workshop_tutorial get_SequenceDNN() ValueError

[beaunorgeot]

I have been walking through the workshop on my own machine. I have changed nothing and am simply running each cell successively.

process:
conda install -c kundajelab dragonn
cd dragonn
python setup.py install
cd examples
ipython notebook ...
{run each cell}
imports, gpu registration etc all work fine

code cell 9: one_filter_dragonn = get_SequenceDNN(one_filter_dragonn_parameters)
Fails: ValueError

If I have time this weekend, I'll try to debug this, but I thought someone else might want to look too

Trace:

ValueError Traceback (most recent call last)
in ()
----> 1 one_filter_dragonn = get_SequenceDNN(one_filter_dragonn_parameters)

/home/t-benorg/anaconda/lib/python2.7/site-packages/dragonn/tutorial_utils.pyc in get_SequenceDNN(SequenceDNN_parameters)
80
81 def get_SequenceDNN(SequenceDNN_parameters):
---> 82 return SequenceDNN(**SequenceDNN_parameters)
83
84

/home/t-benorg/anaconda/lib/python2.7/site-packages/dragonn/models.pyc in init(self, seq_length, use_deep_CNN, use_RNN, num_tasks, num_filters, conv_width, num_filters_2, conv_width_2, num_filters_3, conv_width_3, pool_width, L1, dropout, GRU_size, TDD_size, verbose)
129 nb_filter=num_filters, nb_row=4,
130 nb_col=conv_width, activation='linear',
--> 131 init='he_normal', input_shape=self.input_shape))
132 self.model.add(Activation('relu'))
133 self.model.add(Dropout(dropout))

/home/t-benorg/anaconda/lib/python2.7/site-packages/Keras-0.3.2-py2.7.egg/keras/layers/convolutional.pyc in init(self, nb_filter, nb_row, nb_col, init, activation, weights, border_mode, subsample, dim_ordering, W_regularizer, b_regularizer, activity_regularizer, W_constraint, b_constraint, **kwargs)
253 self.initial_weights = weights
254 self.input = K.placeholder(ndim=4)
--> 255 super(Convolution2D, self).init(**kwargs)
256
257 def build(self):

/home/t-benorg/anaconda/lib/python2.7/site-packages/Keras-0.3.2-py2.7.egg/keras/layers/core.pyc in init(self, **kwargs)
49 self.set_input_shape(tuple(kwargs['batch_input_shape']))
50 elif 'input_shape' in kwargs:
---> 51 self.set_input_shape((None,) + tuple(kwargs['input_shape']))
52 self.trainable = True
53 if 'trainable' in kwargs:

/home/t-benorg/anaconda/lib/python2.7/site-packages/Keras-0.3.2-py2.7.egg/keras/layers/core.pyc in set_input_shape(self, input_shape)
155 self._input_shape = input_shape
156 self.input = K.placeholder(shape=self._input_shape)
--> 157 self.build()
158
159 @Property

/home/t-benorg/anaconda/lib/python2.7/site-packages/Keras-0.3.2-py2.7.egg/keras/layers/convolutional.pyc in build(self)
264 else:
265 raise Exception('Invalid dim_ordering: ' + self.dim_ordering)
--> 266 self.W = self.init(self.W_shape)
267 self.b = K.zeros((self.nb_filter,))
268 self.trainable_weights = [self.W, self.b]

/home/t-benorg/anaconda/lib/python2.7/site-packages/Keras-0.3.2-py2.7.egg/keras/initializations.pyc in he_normal(shape, name)
46 ''' Reference: He et al., http://arxiv.org/abs/1502.01852
47 '''
---> 48 fan_in, fan_out = get_fans(shape)
49 s = np.sqrt(2. / fan_in)
50 return normal(shape, s, name=name)

/home/t-benorg/anaconda/lib/python2.7/site-packages/Keras-0.3.2-py2.7.egg/keras/initializations.pyc in get_fans(shape)
5
6 def get_fans(shape):
----> 7 fan_in = shape[0] if len(shape) == 2 else np.prod(shape[1:])
8 fan_out = shape[1] if len(shape) == 2 else shape[0]
9 return fan_in, fan_out

/home/t-benorg/anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in prod(a, axis, dtype, out, keepdims)
2490 except AttributeError:
2491 return _methods._prod(a, axis=axis, dtype=dtype,
-> 2492 out=out, keepdims=keepdims)
2493 return prod(axis=axis, dtype=dtype, out=out)
2494 else:

/home/t-benorg/anaconda/lib/python2.7/site-packages/numpy/core/_methods.pyc in _prod(a, axis, dtype, out, keepdims)
33
34 def _prod(a, axis=None, dtype=None, out=None, keepdims=False):
---> 35 return umr_prod(a, axis, dtype, out, keepdims)
36
37 def _any(a, axis=None, dtype=None, out=None, keepdims=False):

ValueError: setting an array element with a sequence.

[jisraeli]

It looks like notebook tutorial is from 0.1.1 release used at the ucsf workshop while the installed dragonn package is the 0.1.0 release. Can you try updating the package by running Python setup.py install?

I haven't updated the conda package yet so it comes with 0.1.0 at the moment. I'll update it to 0.1.1 tonight.

[beaunorgeot]

Yea, I'd tried that. Retried with the same failure. The setup.py that I'm using is trying to install the older dragonn version.

From install log

creating 'dist/dragonn-0.1.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dragonn-0.1.0-py2.7.egg
Removing /home/t-benorg/anaconda/lib/python2.7/site-packages/dragonn-0.1.0-py2.7.egg
Copying dragonn-0.1.0-py2.7.egg to /home/t-benorg/anaconda/lib/python2.7/site-packages
dragonn 0.1.0 is already the active version in easy-install.pth
Installing dragonn script to /home/t-benorg/anaconda/bin

[jisraeli]

Looks like I forgot to update setup.py - it still points to 0.1.0. can you try changing to 0.1.1 and reinstalling?

[beaunorgeot]

pointed setup to 0.1.1 and re-ran setup.py. Exited shell. Started new shell and opened notebook.

During initial import I see:
UserWarning: Module dragonn was already imported from /home/t-benorg/anaconda/lib/python2.7/site-packages/dragonn/init.pyc, but /home/t-benorg/dragonn/examples is being added to sys.path

one_filter_dragonn = get_SequenceDNN(one_filter_dragonn_parameters) still fails with same error

[jisraeli]

@beaunorgeot: could you share how you fixed this issue?

[beaunorgeot]

I wasn't able to fix this issue directly. But I did find a simple work around which was to pass arguments directly to DNN():
dnn = get_SequenceDNN(seq_length = 500, num_filters = [16], conv_width=[10], pool_width = 5, dropout = 0.2)

[jisraeli]

I see. If I have time next week I'll try to reproduce this error with fresh install, otherwise we will just update the notebook tutorial for the next release (it will provide a more flexible model specification format). I'll let you know when we have an update on this.

[jisraeli]

@beaunorgeot: we pushed a bunch of updates since the UCSF workshop and haven't seen this issue come up recently. So I'm going to close this, but feel free to reopen this issue if it comes up again.

[bvernot]

@jisraeli - I'm encountering the same "ValueError: setting an array element with a sequence" problem, using dragonn version 0.1.3 (installed yesterday). The previous fix (to provide parameters directly to get_SequenceDNN) doesn't work:

TypeError: get_SequenceDNN() got an unexpected keyword argument 'seq_length'

Any thoughts?

[jisraeli]

@bvernot: thanks for reporting this issue. I'm on vacation this week, I will look through this next week.

[bvernot]

Any updates/thoughts?

[jisraeli]

@bvernot: get_SequenceDNN initializes a SequenceDNN. In versions 0.1.2 and the bleeding edge, there is a seq_length argument. In previous versions that argument is not present. This suggests that an older version of the code is being run. Can you try version 0.1.2 and see if you get the same error?