【推荐系统】：Deep Crossing模型解析以及代码实现

Deep Crossing模型是由微软提出，在微软的搜索引擎bing的搜索广告场景当中，用户除了会返回相关的结果，还会返回相应的广告，因此尽可能的增加广告的点击率，是微软所考虑的重中之重。

因此才设计出了Deep Crossing模型来解决这个问题。这个模型的结构如下所示：

 最下面的各种feature是我们输入层，表示了针对特定的应用背景，微软使用的特征如下：

• Query（搜索）：用户搜索的关键词；
• Keyword（广告关键词）：广告商对自己的产品广告打的标签，用于匹配用户的搜索词；
• Title（标题）：广告的标题；
• Landing Page（落地网页）：点击广告后跳转的网页；
• Match Type（匹配类型）：广告商可选择的关键字与用户查询的匹配程度，通常有四种：精确匹配、短语匹配、宽泛匹配和上下文相关匹配；
• Campaign（广告计划）：广告商投放的计划；
• Imression（曝光样例）：记录了该广告实际曝光场景的相关信息；
• Click（点击样例）：记录了该广告实际点击场景的相关信息；
• Click Through Rate（点击率）：广告的历史点击率
• click prediction（预估点击率）：另一个CTR模型的预估值；

Embedding层

几乎所有基于深度学习的推荐、CTR预估模型都离不开Embedding层，它的作用是将离散高维的稀疏特征转化为低维的密集型特征。Embedding矩阵的参数通过神经网络的反向传播进行训练。在模型结构中发现Feature #2并没有使用Embedding，因为文章提到“维度小于256的特征“不需要进行Embedding转化。

Stacking层

Stacking层的工作特别简单，就是将所有的Embedding向量、或者未进行Embedding操作的原生特征进行拼接。

Multiple Residual Units层

Deep Crossing模型中的Crossing就是多个残差单元层来实现。该层使用了残差网络的基本单元。也就是我们通常所说的ResNet.

Scoring层

Scoring层就也是输出层。一般情况下对于CTR预估模型，往往是一个二分类问题，因此采用逻辑回归来对点击进行预测，正好逻辑回归模型将我们的CTR,也就是点击率，投射到一个从0-1的空间内，形成一个概率，其意义正好和CTR相同。如果要考虑是一个多分类问题，则可以使用softmax进行预测。

代码用tensorflow2实现如下：

1.model.py

import tensorflow as tf
from tensorflow.keras import Model
from tensorflow.keras.regularizers import l2
from tensorflow.keras.layers import Embedding, Dense, Dropout, Input

from modules import Residual_Units


class Deep_Crossing(Model):
    def __init__(self, feature_columns, hidden_units, res_dropout=0., embed_reg=1e-6):
        """
        Deep&Crossing
        :param feature_columns: A list. sparse column feature information.
        :param hidden_units: A list. Neural network hidden units.
        :param res_dropout: A scalar. Dropout of resnet.
        :param embed_reg: A scalar. The regularizer of embedding.
        """
        super(Deep_Crossing, self).__init__()
        self.sparse_feature_columns = feature_columns
        self.embed_layers = {
            'embed_' + str(i): Embedding(input_dim=feat['feat_num'],
                                         input_length=1,
                                         output_dim=feat['embed_dim'],
                                         embeddings_initializer='random_uniform',
                                         embeddings_regularizer=l2(embed_reg))
            for i, feat in enumerate(self.sparse_feature_columns)
        }
        # the total length of embedding layers
        embed_layers_len = sum([feat['embed_dim'] for feat in self.sparse_feature_columns])
        self.res_network = [Residual_Units(unit, embed_layers_len) for unit in hidden_units]
        self.res_dropout = Dropout(res_dropout)
        self.dense = Dense(1, activation=None)

    def call(self, inputs):
        sparse_inputs = inputs
        sparse_embed = tf.concat([self.embed_layers['embed_{}'.format(i)](sparse_inputs[:, i])
                                  for i in range(sparse_inputs.shape[1])], axis=-1)
        r = sparse_embed
        for res in self.res_network:
            r = res(r)
        r = self.res_dropout(r)
        outputs = tf.nn.sigmoid(self.dense(r))
        return outputs

    def summary(self):
        sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)
        Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()

2.modules.py

import tensorflow as tf
from tensorflow.keras.layers import Dense, ReLU, Layer


class Residual_Units(Layer):
    """
    Residual Units
    """
    def __init__(self, hidden_unit, dim_stack):
        """
        :param hidden_unit: A list. Neural network hidden units.
        :param dim_stack: A scalar. The dimension of inputs unit.
        """
        super(Residual_Units, self).__init__()
        self.layer1 = Dense(units=hidden_unit, activation='relu')
        self.layer2 = Dense(units=dim_stack, activation=None)
        self.relu = ReLU()

    def call(self, inputs, **kwargs):
        x = inputs
        x = self.layer1(x)
        x = self.layer2(x)
        outputs = self.relu(x + inputs)
        return outputs

3.train.py 

import tensorflow as tf
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.losses import binary_crossentropy
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import AUC

from model import Deep_Crossing
from data_process.criteo import create_criteo_dataset

import os

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'


if __name__ == '__main__':
    # =============================== GPU ==============================
    # gpu = tf.config.experimental.list_physical_devices(device_type='GPU')
    # print(gpu)
    # If you have GPU, and the value is GPU serial number.
    os.environ['CUDA_VISIBLE_DEVICES'] = '4'
    # ========================= Hyper Parameters =======================
    # you can modify your file path
    file = '../dataset/Criteo/train.txt'
    read_part = True
    sample_num = 5000000
    test_size = 0.2

    embed_dim = 8
    dnn_dropout = 0.5
    hidden_units = [256, 128, 64]

    learning_rate = 0.001
    batch_size = 4096
    epochs = 10

    # ========================== Create dataset =======================
    feature_columns, train, test = create_criteo_dataset(file=file,
                                                         embed_dim=embed_dim,
                                                         read_part=read_part,
                                                         sample_num=sample_num,
                                                         test_size=test_size)
    train_X, train_y = train
    test_X, test_y = test
    # ============================Build Model==========================
    mirrored_strategy = tf.distribute.MirroredStrategy()
    with mirrored_strategy.scope():
        model = Deep_Crossing(feature_columns, hidden_units)
        model.summary()
        # =========================Compile============================
        model.compile(loss=binary_crossentropy, optimizer=Adam(learning_rate=learning_rate),
                      metrics=[AUC()])
    # ============================model checkpoint======================
    # check_path = 'save/deep_crossing_weights.epoch_{epoch:04d}.val_loss_{val_loss:.4f}.ckpt'
    # checkpoint = tf.keras.callbacks.ModelCheckpoint(check_path, save_weights_only=True,
    #                                                 verbose=1, period=5)
    # ===========================Fit==============================
    model.fit(
        train_X,
        train_y,
        epochs=epochs,
        callbacks=[EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)],  # checkpoint
        batch_size=batch_size,
        validation_split=0.1
    )
    # ===========================Test==============================
    print('test AUC: %f' % model.evaluate(test_X, test_y, batch_size=batch_size)[1])