论文笔记 [5] Learning a Deep Convolutional Network for Image Super-Resolution
emmm……这篇文章就是在那篇做deblocking和deringing的文章中提到的，仍然是大佬汤晓鸥等做的深度CNN超分辨率的网络，就是SRCNN。这篇文章给出了一个端到端的，进去低分辨率出来高分辨率。并且说明了传统的sparse coding的SR方法也可以用CNN表示。而且这个网络比较轻量，效果好且速度快。
超分辨率实际上是一个很经典的问题，有的方法是通过图片内在的自相似性，而有的是通过external low- and high-resolution exemplar pairs 来学习映射关系。sparse coding（稀疏编码）是representative methods for external example-based image super-resolution 的一种方法。SC方法是这样，先分patch并且预处理，然后将这些patch用low-resolution dict编码，得到稀疏的coefficient，然后把码本换掉，换成high-resolution dict，然后重构出高分辨率的图像。所以以往对于基于SC的方法，人们主要关注怎么找到最好的码本，或其他模型。
CNN解决SR问题，不必显式地学习dict，manifold，modeling patch space 等等，而是用隐含层隐式学到。而且几乎不怎么用预处理和后处理。
SRCNN
SRCNN有以下几个操作：
- patch extraction and representation
- non-linear mapping
- reconstruction
如下图所示：
patch extraction and representation 即一个可以产生n1个feature map的卷积层。non-linear mapping 环节，我们希望将每一个n1维的向量映射到n2维的向量，可以用1×1的kernel实现。文章中说：It is possible to add more convolutional layers (whose spatial supports are 1 × 1) to increase the non-linearity. But this can significantly increase the complexity of the model, and thus demands more training data and time. 虽然可以增加非线性（？），但是会使得模型变得复杂，所以需要更多的数据和训练时间。因此文章只用了一层map层。最后一层卷积回原来的通道数量c，三层的公式为：
Relationship to Sparse-Coding based Method
SC based 方法做SR问题的基本思路就是先对low-resolution（LR）的图像取patch并归一化，然后投影到一个LR 的dictionary，然后得到系数，再用 HR 的码本编码回去。在CNN中，第一层相当于提出了码本，filters就是dict中的元素，然后通过非线性映射相当于sparse coding solver，因为在SC的方法中，得到了n1个系数以后，要用sparse coding solver把n1个系数投射到n2系数，一般在SC中，n1=n2 。然后reconstruction过程相当于高分辨率码本进行合成。
文章说：Our non-linear operator can be considered as a pixel-wise fully-connected layer 。因为是1×1的kernel，实际上就是通道间逐像素的fc层。对于SC，没有对每个步骤都优化，而 But not all operations have been considered in the optimization in the sparse-coding-based SR methods. On the contrary, in our convolutional neural network, the low-resolution dictionary, high-resolution dictionary, non-linear mapping, together with mean subtraction and averaging, are all involved in the filters to be optimized. 所以可以对每个步骤达到最优。
通过和SC的对比可以用来调参，如下：
Others
loss function 就是MSE，用MSE实际上是favors a high PSNR，由于公式可以看出，PSNR和MSE的关系。另外PSNR只是部分的与perceptual quality相关，所以如果有更好的可导的loss function，可以在这个框架下把MSE替换掉，这也是传统方法不及之处。
训练用了91张图，然后用set5和set14分别用来对不同的upscaling factor做evaluation。
图像的合成：To synthesize the low-resolution samples {Y i }, we blur a sub-image by a proper Gaussian kernel, sub-sample it by the upscaling factor, and upscale it by the same factor via bicubic interpolation. 训练的patch文章中叫做sub-image，因为不像patch那样需要overlap和average。we mean these samples are treated as small “images” rather than “patches”, in the sense that “patches” are overlapping and require some averaging as post-processing but “sub-images” need not. 这些sub-image都是32×32 。
Following [20], we only consider the luminance channel (in YCrCb color space) in our experiments, so c = 1 in the first/last layer. The two chrominance channels are bicubic upsampled only for the purpose of displaying, but not for training/testing.
CNN模型可以处理多通道，作者说是为了fair comparison with 之前的SC方法，所以只用了一个luminance channel。为了避免boarder effect，没用padding，最后出来的patch是20×20.
关于学习率：We empirically find that a smaller learning rate in the last layer is important for the network to converge (similar to the denoising case [12])
在ImageNet上训练得到了更好的结果。
关于filter number，就是feature map的数量，用更多的feature map会提高performance，但是如果对速度有要求则应该用少一些的filters，也可以取得不错的效果。 关于filter size，This suggests that a reasonably larger filter size could
grasp richer structural information, which in turn lead to better results. However, the deployment speed will also decrease with a larger filter size. Therefore, the choice of the network scale should always be a trade-off between performance and speed. 大尺寸的filter使得效果略略好一些。
与各种传统方法的对比图放一张，看上去貌似在PSNR高到一定程度的情况下，实际上PSNR的少量偏差和 visual / perception 得到的图像质量的偏差并不再是完全同步或等价了，因为HVS对于不同的细节和位置等敏感程度并不是完全一样的，而这一点在PSNR中并未体现。
reference：
Dong, Chao, Chen Change Loy, Kaiming He和Xiaoou Tang. 《Learning a Deep Convolutional Network for Image Super-Resolution》. 收入 Computer Vision – ECCV 2014, 184–99. Lecture Notes in Computer Science. Springer, Cham, 2014. https://doi.org/10.1007/978-3-319-10593-2_13.
2018/01/24
世人个个学长年，不悟常年在目前。我得宛秋平易法，只将食粥致神仙。 —— 陆游