caffe lstm

本文官方链接https://www.cnblogs.com/yanghailin/p/15603194.html，未经授权勿转载
理解 LSTM 网络 https://www.jianshu.com/p/9dc9f41f0b29
Understanding LSTM Networks https://colah.github.io/posts/2015-08-Understanding-LSTMs/
pytorch https://pytorch.org/docs/1.0.1/nn.html?highlight=lstm#torch.nn.LSTM

https://blog.csdn.net/nfzhlk/article/details/86507635
忘记门：
作用对象：细胞状态
作用：将细胞状态中的信息选择性的遗忘
让我们回到语言模型的例子中来基于已经看到的预测下一个词。在这个问题中，细胞状态可能包含当前主语的类别，因此正确的代词可以被选择出来。当我们看到新的主语，我们希望忘记旧的主语。
例如，他今天有事，所以我。。。当处理到‘’我‘’的时候选择性的忘记前面的’他’，或者说减小这个词对后面词的作用。

输入层门：
作用对象：细胞状态
作用：将新的信息选择性的记录到细胞状态中
在我们语言模型的例子中，我们希望增加新的主语的类别到细胞状态中，来替代旧的需要忘记的主语。
例如：他今天有事，所以我。。。。当处理到‘’我‘’这个词的时候，就会把主语我更新到细胞中去。

输出层门：
作用对象：隐层ht
在语言模型的例子中，因为他就看到了一个 代词，可能需要输出与一个 动词 相关的信息。例如，可能输出是否代词是单数还是负数，这样如果是动词的话，我们也知道动词需要进行的词形变化。
例如：上面的例子，当处理到‘’我‘’这个词的时候，可以预测下一个词，是动词的可能性较大，而且是第一人称。
会把前面的信息保存到隐层中去。

caffe lstm的实现方式过于复杂，lstm层继承于RecurrentLayer，然后在该层自建一个net。就是说一个lstm层内部就是一个网络！

上图是一个双向lstm，以右边的lstm1X_r2为例。

layer {
    name: "lstm1x_r2"
    type: "LSTM"
    bottom: "permuted_data"
    bottom: "indicator"
    top: "lstm1x"
    recurrent_param {
        num_output: 100
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
          value: 0
        }
    }
}

输入[80,1,512,8] 经过inner

这里从bottom permuted_data数据流shape是[80,1,512,8], indicator数据流的shape是[80,1],这里indicator80个数据是第一个为0，其余为1.
输入的[80,1,512,8] 在lstm内部叫x，type是input。 对这股数据流进行全连接，inner，得到W_xc_x,其shape是[80,1,400],然后经过slice层被切割成80份，W_xc_x_1[1,1,400] W_xc_x_2[1,1,400] ... W_xc_x_80[1,1,400].
注意这里是一个全连接层，会有2组参数，w和b。w的shape是[400,4096],b的shape是[400]. （512×8=4096）
这里的全连接层和公式结合起来就是wx，设定的lstm的recurrent_param num_output: 100，然后这个的全连接层参数直接是num_output×4=400，就是输出是400.
知道这里为什么是4倍关系吗？
仔细看公式，你会发现是有4组w与x相乘的，没错，就是后面的遗忘门f，输入门i，细胞状态g（c～），输出门o都是w与x相乘，所以这里它们是乘以的不同的x，也就是说为每一个门都配置了可学习参数w。
w的shape是[400,4096]，原本hidden参数设置的是100，现在就是这100里面每一个有4组参数，所以就是400了！
W_xc_x_1，W_xc_x_2，。。。，W_xc_x_80这些shape是[1,1,400]。它们后续与h_t-1相加然后在循环中最为gate_input一个接一个的在LSTMUnitLayer层中做逻辑运算。

c(t) h(t)

然后讲解c0， h0， 这两个初始状态都被置位0，shape都是[1,1,100]。
其中根据公式，h也是需要经过全连接层的，h[1,1,100]经过inner变成[1,1,400]. 所以这里也会产生一组参数w[400,100]. 这里在caffe中偏置是没有的，所以caffe的可以学习参数就是这3组，这里一组还是上面全连接层参数是2组。
还有需要注意的是这里的参数是共享的。因为每个h都会用这同样的参数进行全连接运算。
h0是全0的[1,1,100],后面会循环timestep次产生h1,h2,...,h79.
这里就是对应公式中的w×h。这里也是由100生成400个出来，每个i,f,o,g都是自己的。所以100变成400.

gate_input

然后把这两组全连接之后的值相加，得到gate_input_1[1,1,400]。
W_xc_x_t [1,1,400]
h(t-1)' [1,1,400]
就是上面公式中的
wf[h(t-1),xt]
wi[h(t-1),xt]
wc[h(t-1),xt]
wo[h(t-1),xt]
当然要时刻记住这里其实是100组，每组是4个。每个对应了i,f,o,g。
pytorch手册上面的公式

这里就是相当于公式前4个括号里面的相加部分。
这里再简单说下caffe源码里面的indicator，在层lstm里面就是input，层名叫cont。80个，第一个为0，其余为1.这个与h相乘。

LSTMUnit

然后输入3个参数到LSTMUnit层，该层就是负责逻辑运算的，输入的3个参数分别是c(t-1)shape是[1,1,100], gate_input_t[1,1,400], cont[1,1].
LSTMUnit层输出c(t) [1,1,100], h(t) [1,1,100]
这里附上逻辑运算过程代码：

template <typename Dtype>
void LSTMUnitLayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom,
    const vector<Blob<Dtype>*>& top) {
  const int num = bottom[0]->shape(1);//1
  const int x_dim = hidden_dim_ * 4;
  const Dtype* C_prev = bottom[0]->cpu_data();
  const Dtype* X = bottom[1]->cpu_data();
  const Dtype* cont = bottom[2]->cpu_data();
  Dtype* C = top[0]->mutable_cpu_data();
  Dtype* H = top[1]->mutable_cpu_data();
  for (int n = 0; n < num; ++n) { //1
    for (int d = 0; d < hidden_dim_; ++d) {//100
      const Dtype i = sigmoid(X[d]);
      const Dtype f = (*cont == 0) ? 0 :
          (*cont * sigmoid(X[1 * hidden_dim_ + d]));
      const Dtype o = sigmoid(X[2 * hidden_dim_ + d]);
      const Dtype g = tanh(X[3 * hidden_dim_ + d]);
      const Dtype c_prev = C_prev[d];
      const Dtype c = f * c_prev + i * g;
      C[d] = c;
      const Dtype tanh_c = tanh(c);
      H[d] = o * tanh_c;
    }
    C_prev += hidden_dim_;
    X += x_dim;
    C += hidden_dim_;
    H += hidden_dim_;
    ++cont;
  }
}

从代码中可以看到X[d] X[1 * hidden_dim_ + d] X[2 * hidden_dim_ + d] X[3 * hidden_dim_ + d]，大循环是hidden_dim_，100， 每次拿出i,f,o,g的值进行运算。

这里经过了一个单元运算，总共需要time step次运算，每次都会把3个输入参数先得，输入的3个参数分别是c(t-1)shape是[1,1,100], gate_input_t[1,1,400], cont[1,1].再送到LSTMUnitLayer单元运算。
c(t-1)是上一次的运算输出， gate_input_t里面包含了当前的x和上一次输出的h。
最后结果就是直接把h的输出都统一保存，变成[80,1,100]. 每次得到的h是[1,1,100]

整个计算流图

恩，以上大概是把整个流程讲解完了，但是其实现过程是特别复杂的，这些计算流程都是用网络来实现的。
手画的大概流程图：

就是先用代码把网络给搭建好。
代码搭网络的代码在lstm_layer.cpp和recurrent_layer.cpp两部分中，部分代码如下：

template <typename Dtype>
void LSTMLayer<Dtype>::FillUnrolledNet(NetParameter* net_param) const {
  const int num_output = this->layer_param_.recurrent_param().num_output();
  CHECK_GT(num_output, 0) << "num_output must be positive";
  const FillerParameter& weight_filler =
      this->layer_param_.recurrent_param().weight_filler();
  const FillerParameter& bias_filler =
      this->layer_param_.recurrent_param().bias_filler();

  // Add generic LayerParameter's (without bottoms/tops) of layer types we'll
  // use to save redundant code.
  LayerParameter hidden_param;
  hidden_param.set_type("InnerProduct");
  hidden_param.mutable_inner_product_param()->set_num_output(num_output * 4);
  hidden_param.mutable_inner_product_param()->set_bias_term(false);
  hidden_param.mutable_inner_product_param()->set_axis(2);
  hidden_param.mutable_inner_product_param()->
      mutable_weight_filler()->CopyFrom(weight_filler);

  LayerParameter biased_hidden_param(hidden_param);
  biased_hidden_param.mutable_inner_product_param()->set_bias_term(true);
  biased_hidden_param.mutable_inner_product_param()->
      mutable_bias_filler()->CopyFrom(bias_filler);

  LayerParameter sum_param;
  sum_param.set_type("Eltwise");
  sum_param.mutable_eltwise_param()->set_operation(
      EltwiseParameter_EltwiseOp_SUM);

  LayerParameter scale_param;
  scale_param.set_type("Scale");
  scale_param.mutable_scale_param()->set_axis(0);

  LayerParameter slice_param;
  slice_param.set_type("Slice");
  slice_param.mutable_slice_param()->set_axis(0);

  LayerParameter split_param;
  split_param.set_type("Split");

  vector<BlobShape> input_shapes;
  RecurrentInputShapes(&input_shapes);//1 1 100   1 1 100
  CHECK_EQ(2, input_shapes.size());

  LayerParameter* input_layer_param = net_param->add_layer();
  input_layer_param->set_type("Input");
  InputParameter* input_param = input_layer_param->mutable_input_param();

  input_layer_param->add_top("c_0");
  input_param->add_shape()->CopyFrom(input_shapes[0]);

  input_layer_param->add_top("h_0");
  input_param->add_shape()->CopyFrom(input_shapes[1]);

  LayerParameter* cont_slice_param = net_param->add_layer();
  cont_slice_param->CopyFrom(slice_param);
  cont_slice_param->set_name("cont_slice");
  cont_slice_param->add_bottom("cont");
  cont_slice_param->mutable_slice_param()->set_axis(0);

  // Add layer to transform all timesteps of x to the hidden state dimension.
  //     W_xc_x = W_xc * x + b_c
  {
    LayerParameter* x_transform_param = net_param->add_layer();
    x_transform_param->CopyFrom(biased_hidden_param);
    x_transform_param->set_name("x_transform");
    x_transform_param->add_param()->set_name("W_xc");
    x_transform_param->add_param()->set_name("b_c");
    x_transform_param->add_bottom("x");
    x_transform_param->add_top("W_xc_x");
    x_transform_param->add_propagate_down(true);
  }

  if (this->static_input_) {
    // Add layer to transform x_static to the gate dimension.
    //     W_xc_x_static = W_xc_static * x_static
    LayerParameter* x_static_transform_param = net_param->add_layer();
    x_static_transform_param->CopyFrom(hidden_param);
    x_static_transform_param->mutable_inner_product_param()->set_axis(1);
    x_static_transform_param->set_name("W_xc_x_static");
    x_static_transform_param->add_param()->set_name("W_xc_static");
    x_static_transform_param->add_bottom("x_static");
    x_static_transform_param->add_top("W_xc_x_static_preshape");
    x_static_transform_param->add_propagate_down(true);

    LayerParameter* reshape_param = net_param->add_layer();
    reshape_param->set_type("Reshape");
    BlobShape* new_shape =
         reshape_param->mutable_reshape_param()->mutable_shape();
    new_shape->add_dim(1);  // One timestep.
    // Should infer this->N as the dimension so we can reshape on batch size.
    new_shape->add_dim(-1);
    new_shape->add_dim(
        x_static_transform_param->inner_product_param().num_output());
    reshape_param->set_name("W_xc_x_static_reshape");
    reshape_param->add_bottom("W_xc_x_static_preshape");
    reshape_param->add_top("W_xc_x_static");
  }

  LayerParameter* x_slice_param = net_param->add_layer();
  x_slice_param->CopyFrom(slice_param);
  x_slice_param->add_bottom("W_xc_x");
  x_slice_param->set_name("W_xc_x_slice");

  LayerParameter output_concat_layer;
  output_concat_layer.set_name("h_concat");
  output_concat_layer.set_type("Concat");
  output_concat_layer.add_top("h");
  output_concat_layer.mutable_concat_param()->set_axis(0);

//  int TTT = this->T_;  //80
  for (int t = 1; t <= this->T_; ++t) {
    string tm1s = format_int(t - 1);
    string ts = format_int(t);

    cont_slice_param->add_top("cont_" + ts);
    x_slice_param->add_top("W_xc_x_" + ts);

    // Add layers to flush the hidden state when beginning a new
    // sequence, as indicated by cont_t.
    //     h_conted_{t-1} := cont_t * h_{t-1}
    //
    // Normally, cont_t is binary (i.e., 0 or 1), so:
    //     h_conted_{t-1} := h_{t-1} if cont_t == 1
    //                       0   otherwise
    {
      LayerParameter* cont_h_param = net_param->add_layer();
      cont_h_param->CopyFrom(scale_param);
      cont_h_param->set_name("h_conted_" + tm1s);
      cont_h_param->add_bottom("h_" + tm1s);
      cont_h_param->add_bottom("cont_" + ts);
      cont_h_param->add_top("h_conted_" + tm1s);
    }

    // Add layer to compute
    //     W_hc_h_{t-1} := W_hc * h_conted_{t-1}
    {
      LayerParameter* w_param = net_param->add_layer();
      w_param->CopyFrom(hidden_param);
      w_param->set_name("transform_" + ts);
      w_param->add_param()->set_name("W_hc");
      w_param->add_bottom("h_conted_" + tm1s);
      w_param->add_top("W_hc_h_" + tm1s);
      w_param->mutable_inner_product_param()->set_axis(2);
    }

    // Add the outputs of the linear transformations to compute the gate input.
    //     gate_input_t := W_hc * h_conted_{t-1} + W_xc * x_t + b_c
    //                   = W_hc_h_{t-1} + W_xc_x_t + b_c
    {
      LayerParameter* input_sum_layer = net_param->add_layer();
      input_sum_layer->CopyFrom(sum_param);
      input_sum_layer->set_name("gate_input_" + ts);
      input_sum_layer->add_bottom("W_hc_h_" + tm1s);
      input_sum_layer->add_bottom("W_xc_x_" + ts);
      if (this->static_input_) {
        input_sum_layer->add_bottom("W_xc_x_static");
      }
      input_sum_layer->add_top("gate_input_" + ts);
    }

    // Add LSTMUnit layer to compute the cell & hidden vectors c_t and h_t.
    // Inputs: c_{t-1}, gate_input_t = (i_t, f_t, o_t, g_t), cont_t
    // Outputs: c_t, h_t
    //     [ i_t' ]
    //     [ f_t' ] := gate_input_t
    //     [ o_t' ]
    //     [ g_t' ]
    //         i_t := \sigmoid[i_t']
    //         f_t := \sigmoid[f_t']
    //         o_t := \sigmoid[o_t']
    //         g_t := \tanh[g_t']
    //         c_t := cont_t * (f_t .* c_{t-1}) + (i_t .* g_t)
    //         h_t := o_t .* \tanh[c_t]
    {
      LayerParameter* lstm_unit_param = net_param->add_layer();
      lstm_unit_param->set_type("LSTMUnit");
      lstm_unit_param->add_bottom("c_" + tm1s);
      lstm_unit_param->add_bottom("gate_input_" + ts);
      lstm_unit_param->add_bottom("cont_" + ts);
      lstm_unit_param->add_top("c_" + ts);
      lstm_unit_param->add_top("h_" + ts);
      lstm_unit_param->set_name("unit_" + ts);
    }
    output_concat_layer.add_bottom("h_" + ts);
  }  // for (int t = 1; t <= this->T_; ++t)

  {
    LayerParameter* c_T_copy_param = net_param->add_layer();
    c_T_copy_param->CopyFrom(split_param);
    c_T_copy_param->add_bottom("c_" + format_int(this->T_));
    c_T_copy_param->add_top("c_T");
  }
  net_param->add_layer()->CopyFrom(output_concat_layer);
}

手画的代码循环搭建网络的各个层：

代码自动生成的prototxt

这里我给出它用代码搭建网络生成的prototxt, timestep=80.放到netscope里面，直接像蜘蛛网一样的网络。太复杂。
其实也还好，就是把上面的过程重复了80次而已。咱们把一次搞懂就可以，重复循环的事情交给计算机就可以了。

# Enter your network definition here.
# Use Shift+Enter to update the visualization.
layer {
  name: "lstm2x_r2_"
  type: "Input"
  top: "x"
  top: "cont"
  input_param {
    shape {
      dim: 80
      dim: 1
      dim: 512
      dim: 8
    }
    shape {
      dim: 80
      dim: 1
    }
  }
}
layer {
  name: "lstm2x_r2_"
  type: "Input"
  top: "c_0"
  top: "h_0"
  input_param {
    shape {
      dim: 1
      dim: 1
      dim: 100
    }
    shape {
      dim: 1
      dim: 1
      dim: 100
    }
  }
}
layer {
  name: "lstm2x_r2_cont_slice"
  type: "Slice"
  bottom: "cont"
  top: "cont_1"
  top: "cont_2"
  top: "cont_3"
  top: "cont_4"
  top: "cont_5"
  top: "cont_6"
  top: "cont_7"
  top: "cont_8"
  top: "cont_9"
  top: "cont_10"
  top: "cont_11"
  top: "cont_12"
  top: "cont_13"
  top: "cont_14"
  top: "cont_15"
  top: "cont_16"
  top: "cont_17"
  top: "cont_18"
  top: "cont_19"
  top: "cont_20"
  top: "cont_21"
  top: "cont_22"
  top: "cont_23"
  top: "cont_24"
  top: "cont_25"
  top: "cont_26"
  top: "cont_27"
  top: "cont_28"
  top: "cont_29"
  top: "cont_30"
  top: "cont_31"
  top: "cont_32"
  top: "cont_33"
  top: "cont_34"
  top: "cont_35"
  top: "cont_36"
  top: "cont_37"
  top: "cont_38"
  top: "cont_39"
  top: "cont_40"
  top: "cont_41"
  top: "cont_42"
  top: "cont_43"
  top: "cont_44"
  top: "cont_45"
  top: "cont_46"
  top: "cont_47"
  top: "cont_48"
  top: "cont_49"
  top: "cont_50"
  top: "cont_51"
  top: "cont_52"
  top: "cont_53"
  top: "cont_54"
  top: "cont_55"
  top: "cont_56"
  top: "cont_57"
  top: "cont_58"
  top: "cont_59"
  top: "cont_60"
  top: "cont_61"
  top: "cont_62"
  top: "cont_63"
  top: "cont_64"
  top: "cont_65"
  top: "cont_66"
  top: "cont_67"
  top: "cont_68"
  top: "cont_69"
  top: "cont_70"
  top: "cont_71"
  top: "cont_72"
  top: "cont_73"
  top: "cont_74"
  top: "cont_75"
  top: "cont_76"
  top: "cont_77"
  top: "cont_78"
  top: "cont_79"
  top: "cont_80"
  slice_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_x_transform"
  type: "InnerProduct"
  bottom: "x"
  top: "W_xc_x"
  param {
    name: "W_xc"
  }
  param {
    name: "b_c"
  }
  propagate_down: true
  inner_product_param {
    num_output: 400
    bias_term: true
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
      value: 0
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_W_xc_x_slice"
  type: "Slice"
  bottom: "W_xc_x"
  top: "W_xc_x_1"
  top: "W_xc_x_2"
  top: "W_xc_x_3"
  top: "W_xc_x_4"
  top: "W_xc_x_5"
  top: "W_xc_x_6"
  top: "W_xc_x_7"
  top: "W_xc_x_8"
  top: "W_xc_x_9"
  top: "W_xc_x_10"
  top: "W_xc_x_11"
  top: "W_xc_x_12"
  top: "W_xc_x_13"
  top: "W_xc_x_14"
  top: "W_xc_x_15"
  top: "W_xc_x_16"
  top: "W_xc_x_17"
  top: "W_xc_x_18"
  top: "W_xc_x_19"
  top: "W_xc_x_20"
  top: "W_xc_x_21"
  top: "W_xc_x_22"
  top: "W_xc_x_23"
  top: "W_xc_x_24"
  top: "W_xc_x_25"
  top: "W_xc_x_26"
  top: "W_xc_x_27"
  top: "W_xc_x_28"
  top: "W_xc_x_29"
  top: "W_xc_x_30"
  top: "W_xc_x_31"
  top: "W_xc_x_32"
  top: "W_xc_x_33"
  top: "W_xc_x_34"
  top: "W_xc_x_35"
  top: "W_xc_x_36"
  top: "W_xc_x_37"
  top: "W_xc_x_38"
  top: "W_xc_x_39"
  top: "W_xc_x_40"
  top: "W_xc_x_41"
  top: "W_xc_x_42"
  top: "W_xc_x_43"
  top: "W_xc_x_44"
  top: "W_xc_x_45"
  top: "W_xc_x_46"
  top: "W_xc_x_47"
  top: "W_xc_x_48"
  top: "W_xc_x_49"
  top: "W_xc_x_50"
  top: "W_xc_x_51"
  top: "W_xc_x_52"
  top: "W_xc_x_53"
  top: "W_xc_x_54"
  top: "W_xc_x_55"
  top: "W_xc_x_56"
  top: "W_xc_x_57"
  top: "W_xc_x_58"
  top: "W_xc_x_59"
  top: "W_xc_x_60"
  top: "W_xc_x_61"
  top: "W_xc_x_62"
  top: "W_xc_x_63"
  top: "W_xc_x_64"
  top: "W_xc_x_65"
  top: "W_xc_x_66"
  top: "W_xc_x_67"
  top: "W_xc_x_68"
  top: "W_xc_x_69"
  top: "W_xc_x_70"
  top: "W_xc_x_71"
  top: "W_xc_x_72"
  top: "W_xc_x_73"
  top: "W_xc_x_74"
  top: "W_xc_x_75"
  top: "W_xc_x_76"
  top: "W_xc_x_77"
  top: "W_xc_x_78"
  top: "W_xc_x_79"
  top: "W_xc_x_80"
  slice_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_h_conted_0"
  type: "Scale"
  bottom: "h_0"
  bottom: "cont_1"
  top: "h_conted_0"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_1"
  type: "InnerProduct"
  bottom: "h_conted_0"
  top: "W_hc_h_0"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_1"
  type: "Eltwise"
  bottom: "W_hc_h_0"
  bottom: "W_xc_x_1"
  top: "gate_input_1"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_1"
  type: "LSTMUnit"
  bottom: "c_0"
  bottom: "gate_input_1"
  bottom: "cont_1"
  top: "c_1"
  top: "h_1"
}
layer {
  name: "lstm2x_r2_h_conted_1"
  type: "Scale"
  bottom: "h_1"
  bottom: "cont_2"
  top: "h_conted_1"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_2"
  type: "InnerProduct"
  bottom: "h_conted_1"
  top: "W_hc_h_1"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_2"
  type: "Eltwise"
  bottom: "W_hc_h_1"
  bottom: "W_xc_x_2"
  top: "gate_input_2"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_2"
  type: "LSTMUnit"
  bottom: "c_1"
  bottom: "gate_input_2"
  bottom: "cont_2"
  top: "c_2"
  top: "h_2"
}
layer {
  name: "lstm2x_r2_h_conted_2"
  type: "Scale"
  bottom: "h_2"
  bottom: "cont_3"
  top: "h_conted_2"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_3"
  type: "InnerProduct"
  bottom: "h_conted_2"
  top: "W_hc_h_2"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_3"
  type: "Eltwise"
  bottom: "W_hc_h_2"
  bottom: "W_xc_x_3"
  top: "gate_input_3"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_3"
  type: "LSTMUnit"
  bottom: "c_2"
  bottom: "gate_input_3"
  bottom: "cont_3"
  top: "c_3"
  top: "h_3"
}
layer {
  name: "lstm2x_r2_h_conted_3"
  type: "Scale"
  bottom: "h_3"
  bottom: "cont_4"
  top: "h_conted_3"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_4"
  type: "InnerProduct"
  bottom: "h_conted_3"
  top: "W_hc_h_3"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_4"
  type: "Eltwise"
  bottom: "W_hc_h_3"
  bottom: "W_xc_x_4"
  top: "gate_input_4"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_4"
  type: "LSTMUnit"
  bottom: "c_3"
  bottom: "gate_input_4"
  bottom: "cont_4"
  top: "c_4"
  top: "h_4"
}
layer {
  name: "lstm2x_r2_h_conted_4"
  type: "Scale"
  bottom: "h_4"
  bottom: "cont_5"
  top: "h_conted_4"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_5"
  type: "InnerProduct"
  bottom: "h_conted_4"
  top: "W_hc_h_4"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_5"
  type: "Eltwise"
  bottom: "W_hc_h_4"
  bottom: "W_xc_x_5"
  top: "gate_input_5"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_5"
  type: "LSTMUnit"
  bottom: "c_4"
  bottom: "gate_input_5"
  bottom: "cont_5"
  top: "c_5"
  top: "h_5"
}
layer {
  name: "lstm2x_r2_h_conted_5"
  type: "Scale"
  bottom: "h_5"
  bottom: "cont_6"
  top: "h_conted_5"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_6"
  type: "InnerProduct"
  bottom: "h_conted_5"
  top: "W_hc_h_5"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_6"
  type: "Eltwise"
  bottom: "W_hc_h_5"
  bottom: "W_xc_x_6"
  top: "gate_input_6"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_6"
  type: "LSTMUnit"
  bottom: "c_5"
  bottom: "gate_input_6"
  bottom: "cont_6"
  top: "c_6"
  top: "h_6"
}
layer {
  name: "lstm2x_r2_h_conted_6"
  type: "Scale"
  bottom: "h_6"
  bottom: "cont_7"
  top: "h_conted_6"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_7"
  type: "InnerProduct"
  bottom: "h_conted_6"
  top: "W_hc_h_6"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_7"
  type: "Eltwise"
  bottom: "W_hc_h_6"
  bottom: "W_xc_x_7"
  top: "gate_input_7"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_7"
  type: "LSTMUnit"
  bottom: "c_6"
  bottom: "gate_input_7"
  bottom: "cont_7"
  top: "c_7"
  top: "h_7"
}
layer {
  name: "lstm2x_r2_h_conted_7"
  type: "Scale"
  bottom: "h_7"
  bottom: "cont_8"
  top: "h_conted_7"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_8"
  type: "InnerProduct"
  bottom: "h_conted_7"
  top: "W_hc_h_7"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_8"
  type: "Eltwise"
  bottom: "W_hc_h_7"
  bottom: "W_xc_x_8"
  top: "gate_input_8"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_8"
  type: "LSTMUnit"
  bottom: "c_7"
  bottom: "gate_input_8"
  bottom: "cont_8"
  top: "c_8"
  top: "h_8"
}
layer {
  name: "lstm2x_r2_h_conted_8"
  type: "Scale"
  bottom: "h_8"
  bottom: "cont_9"
  top: "h_conted_8"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_9"
  type: "InnerProduct"
  bottom: "h_conted_8"
  top: "W_hc_h_8"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_9"
  type: "Eltwise"
  bottom: "W_hc_h_8"
  bottom: "W_xc_x_9"
  top: "gate_input_9"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_9"
  type: "LSTMUnit"
  bottom: "c_8"
  bottom: "gate_input_9"
  bottom: "cont_9"
  top: "c_9"
  top: "h_9"
}
layer {
  name: "lstm2x_r2_h_conted_9"
  type: "Scale"
  bottom: "h_9"
  bottom: "cont_10"
  top: "h_conted_9"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_10"
  type: "InnerProduct"
  bottom: "h_conted_9"
  top: "W_hc_h_9"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_10"
  type: "Eltwise"
  bottom: "W_hc_h_9"
  bottom: "W_xc_x_10"
  top: "gate_input_10"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_10"
  type: "LSTMUnit"
  bottom: "c_9"
  bottom: "gate_input_10"
  bottom: "cont_10"
  top: "c_10"
  top: "h_10"
}
layer {
  name: "lstm2x_r2_h_conted_10"
  type: "Scale"
  bottom: "h_10"
  bottom: "cont_11"
  top: "h_conted_10"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_11"
  type: "InnerProduct"
  bottom: "h_conted_10"
  top: "W_hc_h_10"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_11"
  type: "Eltwise"
  bottom: "W_hc_h_10"
  bottom: "W_xc_x_11"
  top: "gate_input_11"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_11"
  type: "LSTMUnit"
  bottom: "c_10"
  bottom: "gate_input_11"
  bottom: "cont_11"
  top: "c_11"
  top: "h_11"
}
layer {
  name: "lstm2x_r2_h_conted_11"
  type: "Scale"
  bottom: "h_11"
  bottom: "cont_12"
  top: "h_conted_11"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_12"
  type: "InnerProduct"
  bottom: "h_conted_11"
  top: "W_hc_h_11"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_12"
  type: "Eltwise"
  bottom: "W_hc_h_11"
  bottom: "W_xc_x_12"
  top: "gate_input_12"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_12"
  type: "LSTMUnit"
  bottom: "c_11"
  bottom: "gate_input_12"
  bottom: "cont_12"
  top: "c_12"
  top: "h_12"
}
layer {
  name: "lstm2x_r2_h_conted_12"
  type: "Scale"
  bottom: "h_12"
  bottom: "cont_13"
  top: "h_conted_12"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_13"
  type: "InnerProduct"
  bottom: "h_conted_12"
  top: "W_hc_h_12"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_13"
  type: "Eltwise"
  bottom: "W_hc_h_12"
  bottom: "W_xc_x_13"
  top: "gate_input_13"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_13"
  type: "LSTMUnit"
  bottom: "c_12"
  bottom: "gate_input_13"
  bottom: "cont_13"
  top: "c_13"
  top: "h_13"
}
layer {
  name: "lstm2x_r2_h_conted_13"
  type: "Scale"
  bottom: "h_13"
  bottom: "cont_14"
  top: "h_conted_13"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_14"
  type: "InnerProduct"
  bottom: "h_conted_13"
  top: "W_hc_h_13"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_14"
  type: "Eltwise"
  bottom: "W_hc_h_13"
  bottom: "W_xc_x_14"
  top: "gate_input_14"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_14"
  type: "LSTMUnit"
  bottom: "c_13"
  bottom: "gate_input_14"
  bottom: "cont_14"
  top: "c_14"
  top: "h_14"
}
layer {
  name: "lstm2x_r2_h_conted_14"
  type: "Scale"
  bottom: "h_14"
  bottom: "cont_15"
  top: "h_conted_14"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_15"
  type: "InnerProduct"
  bottom: "h_conted_14"
  top: "W_hc_h_14"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_15"
  type: "Eltwise"
  bottom: "W_hc_h_14"
  bottom: "W_xc_x_15"
  top: "gate_input_15"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_15"
  type: "LSTMUnit"
  bottom: "c_14"
  bottom: "gate_input_15"
  bottom: "cont_15"
  top: "c_15"
  top: "h_15"
}
layer {
  name: "lstm2x_r2_h_conted_15"
  type: "Scale"
  bottom: "h_15"
  bottom: "cont_16"
  top: "h_conted_15"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_16"
  type: "InnerProduct"
  bottom: "h_conted_15"
  top: "W_hc_h_15"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_16"
  type: "Eltwise"
  bottom: "W_hc_h_15"
  bottom: "W_xc_x_16"
  top: "gate_input_16"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_16"
  type: "LSTMUnit"
  bottom: "c_15"
  bottom: "gate_input_16"
  bottom: "cont_16"
  top: "c_16"
  top: "h_16"
}
layer {
  name: "lstm2x_r2_h_conted_16"
  type: "Scale"
  bottom: "h_16"
  bottom: "cont_17"
  top: "h_conted_16"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_17"
  type: "InnerProduct"
  bottom: "h_conted_16"
  top: "W_hc_h_16"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_17"
  type: "Eltwise"
  bottom: "W_hc_h_16"
  bottom: "W_xc_x_17"
  top: "gate_input_17"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_17"
  type: "LSTMUnit"
  bottom: "c_16"
  bottom: "gate_input_17"
  bottom: "cont_17"
  top: "c_17"
  top: "h_17"
}
layer {
  name: "lstm2x_r2_h_conted_17"
  type: "Scale"
  bottom: "h_17"
  bottom: "cont_18"
  top: "h_conted_17"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_18"
  type: "InnerProduct"
  bottom: "h_conted_17"
  top: "W_hc_h_17"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_18"
  type: "Eltwise"
  bottom: "W_hc_h_17"
  bottom: "W_xc_x_18"
  top: "gate_input_18"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_18"
  type: "LSTMUnit"
  bottom: "c_17"
  bottom: "gate_input_18"
  bottom: "cont_18"
  top: "c_18"
  top: "h_18"
}
layer {
  name: "lstm2x_r2_h_conted_18"
  type: "Scale"
  bottom: "h_18"
  bottom: "cont_19"
  top: "h_conted_18"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_19"
  type: "InnerProduct"
  bottom: "h_conted_18"
  top: "W_hc_h_18"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_19"
  type: "Eltwise"
  bottom: "W_hc_h_18"
  bottom: "W_xc_x_19"
  top: "gate_input_19"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_19"
  type: "LSTMUnit"
  bottom: "c_18"
  bottom: "gate_input_19"
  bottom: "cont_19"
  top: "c_19"
  top: "h_19"
}
layer {
  name: "lstm2x_r2_h_conted_19"
  type: "Scale"
  bottom: "h_19"
  bottom: "cont_20"
  top: "h_conted_19"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_20"
  type: "InnerProduct"
  bottom: "h_conted_19"
  top: "W_hc_h_19"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_20"
  type: "Eltwise"
  bottom: "W_hc_h_19"
  bottom: "W_xc_x_20"
  top: "gate_input_20"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_20"
  type: "LSTMUnit"
  bottom: "c_19"
  bottom: "gate_input_20"
  bottom: "cont_20"
  top: "c_20"
  top: "h_20"
}
layer {
  name: "lstm2x_r2_h_conted_20"
  type: "Scale"
  bottom: "h_20"
  bottom: "cont_21"
  top: "h_conted_20"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_21"
  type: "InnerProduct"
  bottom: "h_conted_20"
  top: "W_hc_h_20"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_21"
  type: "Eltwise"
  bottom: "W_hc_h_20"
  bottom: "W_xc_x_21"
  top: "gate_input_21"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_21"
  type: "LSTMUnit"
  bottom: "c_20"
  bottom: "gate_input_21"
  bottom: "cont_21"
  top: "c_21"
  top: "h_21"
}
layer {
  name: "lstm2x_r2_h_conted_21"
  type: "Scale"
  bottom: "h_21"
  bottom: "cont_22"
  top: "h_conted_21"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_22"
  type: "InnerProduct"
  bottom: "h_conted_21"
  top: "W_hc_h_21"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_22"
  type: "Eltwise"
  bottom: "W_hc_h_21"
  bottom: "W_xc_x_22"
  top: "gate_input_22"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_22"
  type: "LSTMUnit"
  bottom: "c_21"
  bottom: "gate_input_22"
  bottom: "cont_22"
  top: "c_22"
  top: "h_22"
}
layer {
  name: "lstm2x_r2_h_conted_22"
  type: "Scale"
  bottom: "h_22"
  bottom: "cont_23"
  top: "h_conted_22"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_23"
  type: "InnerProduct"
  bottom: "h_conted_22"
  top: "W_hc_h_22"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_23"
  type: "Eltwise"
  bottom: "W_hc_h_22"
  bottom: "W_xc_x_23"
  top: "gate_input_23"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_23"
  type: "LSTMUnit"
  bottom: "c_22"
  bottom: "gate_input_23"
  bottom: "cont_23"
  top: "c_23"
  top: "h_23"
}
layer {
  name: "lstm2x_r2_h_conted_23"
  type: "Scale"
  bottom: "h_23"
  bottom: "cont_24"
  top: "h_conted_23"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_24"
  type: "InnerProduct"
  bottom: "h_conted_23"
  top: "W_hc_h_23"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_24"
  type: "Eltwise"
  bottom: "W_hc_h_23"
  bottom: "W_xc_x_24"
  top: "gate_input_24"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_24"
  type: "LSTMUnit"
  bottom: "c_23"
  bottom: "gate_input_24"
  bottom: "cont_24"
  top: "c_24"
  top: "h_24"
}
layer {
  name: "lstm2x_r2_h_conted_24"
  type: "Scale"
  bottom: "h_24"
  bottom: "cont_25"
  top: "h_conted_24"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_25"
  type: "InnerProduct"
  bottom: "h_conted_24"
  top: "W_hc_h_24"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_25"
  type: "Eltwise"
  bottom: "W_hc_h_24"
  bottom: "W_xc_x_25"
  top: "gate_input_25"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_25"
  type: "LSTMUnit"
  bottom: "c_24"
  bottom: "gate_input_25"
  bottom: "cont_25"
  top: "c_25"
  top: "h_25"
}
layer {
  name: "lstm2x_r2_h_conted_25"
  type: "Scale"
  bottom: "h_25"
  bottom: "cont_26"
  top: "h_conted_25"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_26"
  type: "InnerProduct"
  bottom: "h_conted_25"
  top: "W_hc_h_25"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_26"
  type: "Eltwise"
  bottom: "W_hc_h_25"
  bottom: "W_xc_x_26"
  top: "gate_input_26"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_26"
  type: "LSTMUnit"
  bottom: "c_25"
  bottom: "gate_input_26"
  bottom: "cont_26"
  top: "c_26"
  top: "h_26"
}
layer {
  name: "lstm2x_r2_h_conted_26"
  type: "Scale"
  bottom: "h_26"
  bottom: "cont_27"
  top: "h_conted_26"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_27"
  type: "InnerProduct"
  bottom: "h_conted_26"
  top: "W_hc_h_26"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_27"
  type: "Eltwise"
  bottom: "W_hc_h_26"
  bottom: "W_xc_x_27"
  top: "gate_input_27"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_27"
  type: "LSTMUnit"
  bottom: "c_26"
  bottom: "gate_input_27"
  bottom: "cont_27"
  top: "c_27"
  top: "h_27"
}
layer {
  name: "lstm2x_r2_h_conted_27"
  type: "Scale"
  bottom: "h_27"
  bottom: "cont_28"
  top: "h_conted_27"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_28"
  type: "InnerProduct"
  bottom: "h_conted_27"
  top: "W_hc_h_27"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_28"
  type: "Eltwise"
  bottom: "W_hc_h_27"
  bottom: "W_xc_x_28"
  top: "gate_input_28"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_28"
  type: "LSTMUnit"
  bottom: "c_27"
  bottom: "gate_input_28"
  bottom: "cont_28"
  top: "c_28"
  top: "h_28"
}
layer {
  name: "lstm2x_r2_h_conted_28"
  type: "Scale"
  bottom: "h_28"
  bottom: "cont_29"
  top: "h_conted_28"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_29"
  type: "InnerProduct"
  bottom: "h_conted_28"
  top: "W_hc_h_28"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_29"
  type: "Eltwise"
  bottom: "W_hc_h_28"
  bottom: "W_xc_x_29"
  top: "gate_input_29"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_29"
  type: "LSTMUnit"
  bottom: "c_28"
  bottom: "gate_input_29"
  bottom: "cont_29"
  top: "c_29"
  top: "h_29"
}
layer {
  name: "lstm2x_r2_h_conted_29"
  type: "Scale"
  bottom: "h_29"
  bottom: "cont_30"
  top: "h_conted_29"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_30"
  type: "InnerProduct"
  bottom: "h_conted_29"
  top: "W_hc_h_29"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_30"
  type: "Eltwise"
  bottom: "W_hc_h_29"
  bottom: "W_xc_x_30"
  top: "gate_input_30"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_30"
  type: "LSTMUnit"
  bottom: "c_29"
  bottom: "gate_input_30"
  bottom: "cont_30"
  top: "c_30"
  top: "h_30"
}
layer {
  name: "lstm2x_r2_h_conted_30"
  type: "Scale"
  bottom: "h_30"
  bottom: "cont_31"
  top: "h_conted_30"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_31"
  type: "InnerProduct"
  bottom: "h_conted_30"
  top: "W_hc_h_30"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_31"
  type: "Eltwise"
  bottom: "W_hc_h_30"
  bottom: "W_xc_x_31"
  top: "gate_input_31"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_31"
  type: "LSTMUnit"
  bottom: "c_30"
  bottom: "gate_input_31"
  bottom: "cont_31"
  top: "c_31"
  top: "h_31"
}
layer {
  name: "lstm2x_r2_h_conted_31"
  type: "Scale"
  bottom: "h_31"
  bottom: "cont_32"
  top: "h_conted_31"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_32"
  type: "InnerProduct"
  bottom: "h_conted_31"
  top: "W_hc_h_31"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_32"
  type: "Eltwise"
  bottom: "W_hc_h_31"
  bottom: "W_xc_x_32"
  top: "gate_input_32"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_32"
  type: "LSTMUnit"
  bottom: "c_31"
  bottom: "gate_input_32"
  bottom: "cont_32"
  top: "c_32"
  top: "h_32"
}
layer {
  name: "lstm2x_r2_h_conted_32"
  type: "Scale"
  bottom: "h_32"
  bottom: "cont_33"
  top: "h_conted_32"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_33"
  type: "InnerProduct"
  bottom: "h_conted_32"
  top: "W_hc_h_32"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_33"
  type: "Eltwise"
  bottom: "W_hc_h_32"
  bottom: "W_xc_x_33"
  top: "gate_input_33"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_33"
  type: "LSTMUnit"
  bottom: "c_32"
  bottom: "gate_input_33"
  bottom: "cont_33"
  top: "c_33"
  top: "h_33"
}
layer {
  name: "lstm2x_r2_h_conted_33"
  type: "Scale"
  bottom: "h_33"
  bottom: "cont_34"
  top: "h_conted_33"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_34"
  type: "InnerProduct"
  bottom: "h_conted_33"
  top: "W_hc_h_33"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_34"
  type: "Eltwise"
  bottom: "W_hc_h_33"
  bottom: "W_xc_x_34"
  top: "gate_input_34"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_34"
  type: "LSTMUnit"
  bottom: "c_33"
  bottom: "gate_input_34"
  bottom: "cont_34"
  top: "c_34"
  top: "h_34"
}
layer {
  name: "lstm2x_r2_h_conted_34"
  type: "Scale"
  bottom: "h_34"
  bottom: "cont_35"
  top: "h_conted_34"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_35"
  type: "InnerProduct"
  bottom: "h_conted_34"
  top: "W_hc_h_34"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_35"
  type: "Eltwise"
  bottom: "W_hc_h_34"
  bottom: "W_xc_x_35"
  top: "gate_input_35"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "lstm2x_r2_unit_35"
  type: "LSTMUnit"
  bottom: "c_34"
  bottom: "gate_input_35"
  bottom: "cont_35"
  top: "c_35"
  top: "h_35"
}
layer {
  name: "lstm2x_r2_h_conted_35"
  type: "Scale"
  bottom: "h_35"
  bottom: "cont_36"
  top: "h_conted_35"
  scale_param {
    axis: 0
  }
}
layer {
  name: "lstm2x_r2_transform_36"
  type: "InnerProduct"
  bottom: "h_conted_35"
  top: "W_hc_h_35"
  param {
    name: "W_hc"
  }
  inner_product_param {
    num_output: 400
    bias_term: false
    weight_filler {
      type: "xavier"
    }
    axis: 2
  }
}
layer {
  name: "lstm2x_r2_gate_input_36"
  type: "Eltwise"
  bottom: "W_hc_h_35"
  bottom: "W_xc_x_36"
  top: "hello"
  }