caffe parse_log.sh

画loss曲线需要用到此shell脚本

#!/bin/bash
# Usage parse_log.sh caffe.log
# It creates the following two text files, each containing a table:
#     caffe.log.test (columns: '#Iters Seconds TestAccuracy TestLoss')
#     caffe.log.train (columns: '#Iters Seconds TrainingLoss LearningRate')


# get the dirname of the script
DIR="$( cd "$(dirname "$0")" ; pwd -P )"


if [ "$#" -lt 1 ]
then
echo "Usage parse_log.sh /path/to/your.log"
exit
fi
LOG=`basename $1`
grep -B 1 'Test ' $1 > aux.txt
grep 'Iteration ' aux.txt | sed  's/.*Iteration ([[:digit:]]*).*/1/g' > aux0.txt
grep 'Test net output #0' aux.txt | awk '{print $11}' > aux1.txt
grep 'Test net output #1' aux.txt | awk '{print $11}' > aux2.txt

# Extracting elapsed seconds
# For extraction of time since this line contains the start time
grep '] Solving ' $1 > aux3.txt
grep 'Testing net' $1 >> aux3.txt
$DIR/extract_seconds.py aux3.txt aux4.txt

# Generating
echo '#Iters Seconds TestAccuracy TestLoss'> $LOG.test
paste aux0.txt aux4.txt aux1.txt aux2.txt | column -t >> $LOG.test
rm aux.txt aux0.txt aux1.txt aux2.txt aux3.txt aux4.txt

# For extraction of time since this line contains the start time
grep '] Solving ' $1 > aux.txt
grep ', loss = ' $1 >> aux.txt
grep 'Iteration ' aux.txt | sed  's/.*Iteration ([[:digit:]]*).*/1/g' > aux0.txt
grep ', loss = ' $1 | awk '{print $9}' > aux1.txt
grep ', lr = ' $1 | awk '{print $9}' > aux2.txt

# Extracting elapsed seconds
$DIR/extract_seconds.py aux.txt aux3.txt

# Generating
echo '#Iters Seconds TrainingLoss LearningRate'> $LOG.train
paste aux0.txt aux3.txt aux1.txt aux2.txt | column -t >> $LOG.train
rm aux.txt aux0.txt aux1.txt aux2.txt  aux3.txt

$1表示第一个参数，即命令行输入的第一个参数，parse_log.sh caffe.log，caffe.log就是第一个参数。

grep是linux搜索命令，grep '] Solving ' $1 > aux.txt就是在第一参数里找] Solving，然后把这个所在行重定向到aux.txt文件。

bnrc@bnrc:~$ grep '] Solving ' /home/bnrc/fast-rcnn/caffe-fast-rcnn/tools/extra/out.log
I0619 20:43:38.261850 23209 solver.cpp:189] Solving deeplab_largeFOV

这是自己在电脑上直接执行的grep命令，可以看到，在文件中查询，会返回这个字符串所在的整个这一行。

grep -B 1 'Test ' $1 > aux.txt ：在第一个参数所在文件中搜索有'Test '的行，显示这些行，并显示第一个有'Test '的行的前1行。如果换成2就是前2行。-A是向后显示。

这个脚本是前面一段是提取.test的日志，后半部分是提取.train的日志，两个都用到了aux.txt这几个临时文件，都把他们生成然后删除了。

-b 在显示符合范本样式的那一行之外，并显示该行之前的内容。

来自: http://man.linuxde.net/grepgrep的-A, -B, -C选项分别可以显示匹配行的后,前,后前多少行内容

-b 在显示符合范本样式的那一行之外，并显示该行之前的内容。

来自: http://man.linuxde.net/grep