============================================================================ MonoPhone Training & Decoding ============================================================================ steps/train_mono.sh --nj 30 --cmd run.pl --mem 4G data/train data/lang exp/mono steps/train_mono.sh: Initializing monophone system. steps/train_mono.sh: Compiling training graphs steps/train_mono.sh: Aligning data equally (pass 0) steps/train_mono.sh: Pass 1 steps/train_mono.sh: Aligning data steps/train_mono.sh: Pass 2 steps/train_mono.sh: Aligning data
....
steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/mono
steps/diagnostic/analyze_alignments.sh: see stats in exp/mono/log/analyze_alignments.log
2 warnings in exp/mono/log/align.*.*.log
exp/mono: nj=30 align prob=-99.15 over 3.12h [retry=0.0%, fail=0.0%] states=144 gauss=986
steps/train_mono.sh: Done training monophone system in exp/mono
tree-info exp/mono/tree tree-info exp/mono/tree fsttablecompose data/lang_test_bg/L_disambig.fst data/lang_test_bg/G.fst fstdeterminizestar --use-log=true fstminimizeencoded fstpushspecial fstisstochastic data/lang_test_bg/tmp/LG.fst -0.00841336 -0.00928521 fstcomposecontext --context-size=1 --central-position=0 --read-disambig-syms=data/lang_test_bg/phones/disambig.int --write-disambig-syms=data/lang_test_bg/tmp/disambig_ilabels_1_0.int data/lang_test_bg/tmp/ilabels_1_0.31072 fstisstochastic data/lang_test_bg/tmp/CLG_1_0.fst -0.00841336 -0.00928521 make-h-transducer --disambig-syms-out=exp/mono/graph/disambig_tid.int --transition-scale=1.0 data/lang_test_bg/tmp/ilabels_1_0 exp/mono/tree exp/mono/final.mdl fstminimizeencoded fstdeterminizestar --use-log=true fsttablecompose exp/mono/graph/Ha.fst data/lang_test_bg/tmp/CLG_1_0.fst fstrmsymbols exp/mono/graph/disambig_tid.int fstrmepslocal fstisstochastic exp/mono/graph/HCLGa.fst 0.000381709 -0.00951555 add-self-loops --self-loop-scale=0.1 --reorder=true exp/mono/final.mdl steps/decode.sh --nj 5 --cmd run.pl --mem 4G exp/mono/graph data/dev exp/mono/decode_dev decode.sh: feature type is delta steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/mono/graph exp/mono/decode_dev steps/diagnostic/analyze_lats.sh: see stats in exp/mono/decode_dev/log/analyze_alignments.log Overall, lattice depth (10,50,90-percentile)=(5,25,121) and mean=56.0 steps/diagnostic/analyze_lats.sh: see stats in exp/mono/decode_dev/log/analyze_lattice_depth_stats.log steps/decode.sh --nj 5 --cmd run.pl --mem 4G exp/mono/graph data/test exp/mono/decode_test decode.sh: feature type is delta steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/mono/graph exp/mono/decode_test steps/diagnostic/analyze_lats.sh: see stats in exp/mono/decode_test/log/analyze_alignments.log Overall, lattice depth (10,50,90-percentile)=(6,27,143) and mean=70.8 steps/diagnostic/analyze_lats.sh: see stats in exp/mono/decode_test/log/analyze_lattice_depth_stats.log
============================================================================ tri1 : Deltas + Delta-Deltas Training & Decoding ============================================================================ steps/align_si.sh --boost-silence 1.25 --nj 30 --cmd run.pl --mem 4G data/train data/lang exp/mono exp/mono_ali steps/align_si.sh: feature type is delta steps/align_si.sh: aligning data in data/train using model from exp/mono, putting alignments in exp/mono_ali steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/mono_ali steps/diagnostic/analyze_alignments.sh: see stats in exp/mono_ali/log/analyze_alignments.log steps/align_si.sh: done aligning data. steps/train_deltas.sh --cmd run.pl --mem 4G 2500 15000 data/train data/lang exp/mono_ali exp/tri1 steps/train_deltas.sh: accumulating tree stats steps/train_deltas.sh: getting questions for tree-building, via clustering steps/train_deltas.sh: building the tree steps/train_deltas.sh: converting alignments from exp/mono_ali to use current tree steps/train_deltas.sh: compiling graphs of transcripts steps/train_deltas.sh: training pass 1 steps/train_deltas.sh: training pass 2
...
steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/tri1
steps/diagnostic/analyze_alignments.sh: see stats in exp/tri1/log/analyze_alignments.log
exp/tri1: nj=30 align prob=-95.28 over 3.12h [retry=0.0%, fail=0.0%] states=1893 gauss=15025 tree-impr=5.40
steps/train_deltas.sh: Done training system with delta+delta-delta features in exp/tri1
steps/train_deltas.sh: Done training system with delta+delta-delta features in exp/tri1
tree-info exp/tri1/tree
tree-info exp/tri1/tree
fstcomposecontext --context-size=3 --central-position=1 --read-disambig-syms=data/lang_test_bg/phones/disambig.int --write-disambig-syms=data/lang_test_bg/tmp/disambig_ilabels_3_1.int data/lang_test_bg/tmp/ilabels_3_1.28346
fstisstochastic data/lang_test_bg/tmp/CLG_3_1.fst
0 -0.00928518
make-h-transducer --disambig-syms-out=exp/tri1/graph/disambig_tid.int --transition-scale=1.0 data/lang_test_bg/tmp/ilabels_3_1 exp/tri1/tree exp/tri1/final.mdl
fsttablecompose exp/tri1/graph/Ha.fst data/lang_test_bg/tmp/CLG_3_1.fst
fstdeterminizestar --use-log=true
fstrmsymbols exp/tri1/graph/disambig_tid.int
fstminimizeencoded
fstrmepslocal
fstisstochastic exp/tri1/graph/HCLGa.fst
0.000449687 -0.0175771
HCLGa is not stochastic
add-self-loops --self-loop-scale=0.1 --reorder=true exp/tri1/final.mdl
steps/decode.sh --nj 5 --cmd run.pl --mem 4G exp/tri1/graph data/dev exp/tri1/decode_dev
decode.sh: feature type is delta
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri1/graph exp/tri1/decode_dev
steps/diagnostic/analyze_lats.sh: see stats in exp/tri1/decode_dev/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(3,11,41) and mean=19.0
steps/diagnostic/analyze_lats.sh: see stats in exp/tri1/decode_dev/log/analyze_lattice_depth_stats.log
steps/decode.sh --nj 5 --cmd run.pl --mem 4G exp/tri1/graph data/test exp/tri1/decode_test
decode.sh: feature type is delta
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri1/graph exp/tri1/decode_test
steps/diagnostic/analyze_lats.sh: see stats in exp/tri1/decode_test/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(3,12,47) and mean=21.8
steps/diagnostic/analyze_lats.sh: see stats in exp/tri1/decode_test/log/analyze_lattice_depth_stats.log
============================================================================ tri2 : LDA + MLLT Training & Decoding ============================================================================ steps/align_si.sh --nj 30 --cmd run.pl --mem 4G data/train data/lang exp/tri1 exp/tri1_ali steps/align_si.sh: feature type is delta steps/align_si.sh: aligning data in data/train using model from exp/tri1, putting alignments in exp/tri1_ali steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/tri1_ali steps/diagnostic/analyze_alignments.sh: see stats in exp/tri1_ali/log/analyze_alignments.log steps/align_si.sh: done aligning data. steps/train_lda_mllt.sh --cmd run.pl --mem 4G --splice-opts --left-context=3 --right-context=3 2500 15000 data/train data/lang exp/tri1_ali exp/tri2 steps/train_lda_mllt.sh: Accumulating LDA statistics. steps/train_lda_mllt.sh: Accumulating tree stats steps/train_lda_mllt.sh: Getting questions for tree clustering. steps/train_lda_mllt.sh: Building the tree steps/train_lda_mllt.sh: Initializing the model steps/train_lda_mllt.sh: Converting alignments from exp/tri1_ali to use current tree steps/train_lda_mllt.sh: Compiling graphs of transcripts Training pass 1 Training pass 2 steps/train_lda_mllt.sh: Estimating MLLT Training pass 3 Training pass 4
...
steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/tri2
steps/diagnostic/analyze_alignments.sh: see stats in exp/tri2/log/analyze_alignments.log
1 warnings in exp/tri2/log/compile_questions.log
110 warnings in exp/tri2/log/update.*.log
99 warnings in exp/tri2/log/init_model.log
exp/tri2: nj=30 align prob=-47.93 over 3.12h [retry=0.0%, fail=0.0%] states=2021 gauss=15026 tree-impr=5.57 lda-sum=28.43 mllt:impr,logdet=1.66,2.28
steps/train_lda_mllt.sh: Done training system with LDA+MLLT features in exp/tri2
tree-info exp/tri2/tree
tree-info exp/tri2/tree
make-h-transducer --disambig-syms-out=exp/tri2/graph/disambig_tid.int --transition-scale=1.0 data/lang_test_bg/tmp/ilabels_3_1 exp/tri2/tree exp/tri2/final.mdl
fsttablecompose exp/tri2/graph/Ha.fst data/lang_test_bg/tmp/CLG_3_1.fst
fstminimizeencoded
fstdeterminizestar --use-log=true
fstrmsymbols exp/tri2/graph/disambig_tid.int
fstrmepslocal
fstisstochastic exp/tri2/graph/HCLGa.fst
0.000472258 -0.0175772
HCLGa is not stochastic
add-self-loops --self-loop-scale=0.1 --reorder=true exp/tri2/final.mdl
steps/decode.sh --nj 5 --cmd run.pl --mem 4G exp/tri2/graph data/dev exp/tri2/decode_dev
decode.sh: feature type is lda
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri2/graph exp/tri2/decode_dev
steps/diagnostic/analyze_lats.sh: see stats in exp/tri2/decode_dev/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(2,8,29) and mean=13.2
steps/diagnostic/analyze_lats.sh: see stats in exp/tri2/decode_dev/log/analyze_lattice_depth_stats.log
steps/decode.sh --nj 5 --cmd run.pl --mem 4G exp/tri2/graph data/test exp/tri2/decode_test
decode.sh: feature type is lda
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri2/graph exp/tri2/decode_test
steps/diagnostic/analyze_lats.sh: see stats in exp/tri2/decode_test/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(2,9,33) and mean=14.9
steps/diagnostic/analyze_lats.sh: see stats in exp/tri2/decode_test/log/analyze_lattice_depth_stats.log
============================================================================ tri3 : LDA + MLLT + SAT Training & Decoding ============================================================================ steps/align_si.sh --nj 30 --cmd run.pl --mem 4G --use-graphs true data/train data/lang exp/tri2 exp/tri2_ali steps/align_si.sh: feature type is lda steps/align_si.sh: aligning data in data/train using model from exp/tri2, putting alignments in exp/tri2_ali steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/tri2_ali steps/diagnostic/analyze_alignments.sh: see stats in exp/tri2_ali/log/analyze_alignments.log steps/align_si.sh: done aligning data. steps/train_sat.sh --cmd run.pl --mem 4G 2500 15000 data/train data/lang exp/tri2_ali exp/tri3 steps/train_sat.sh: feature type is lda steps/train_sat.sh: obtaining initial fMLLR transforms since not present in exp/tri2_ali steps/train_sat.sh: Accumulating tree stats steps/train_sat.sh: Getting questions for tree clustering. steps/train_sat.sh: Building the tree steps/train_sat.sh: Initializing the model steps/train_sat.sh: Converting alignments from exp/tri2_ali to use current tree steps/train_sat.sh: Compiling graphs of transcripts Pass 1 Pass 2 Estimating fMLLR transforms Pass 3 Pass 4
...
steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/tri3
steps/diagnostic/analyze_alignments.sh: see stats in exp/tri3/log/analyze_alignments.log
15 warnings in exp/tri3/log/update.*.log
43 warnings in exp/tri3/log/init_model.log
1 warnings in exp/tri3/log/compile_questions.log
steps/train_sat.sh: Likelihood evolution:
-50.2406 -49.3636 -49.1648 -48.9681 -48.2487 -47.5314 -47.0963 -46.8406 -46.6005 -46.0718 -45.8132 -45.4844 -45.2978 -45.1566 -45.0342 -44.9283 -44.8193 -44.7107 -44.6095 -44.4491 -44.3126 -44.2252 -44.1415 -44.0622 -43.9841 -43.9087 -43.8345 -43.7629 -43.6936 -43.6003 -43.5265 -43.5005 -43.484 -43.4717
exp/tri3: nj=30 align prob=-47.09 over 3.12h [retry=0.0%, fail=0.0%] states=1920 gauss=15011 fmllr-impr=4.04 over 2.79h tree-impr=8.82
steps/train_sat.sh: done training SAT system in exp/tri3
tree-info exp/tri3/tree
tree-info exp/tri3/tree
make-h-transducer --disambig-syms-out=exp/tri3/graph/disambig_tid.int --transition-scale=1.0 data/lang_test_bg/tmp/ilabels_3_1 exp/tri3/tree exp/tri3/final.mdl
fsttablecompose exp/tri3/graph/Ha.fst data/lang_test_bg/tmp/CLG_3_1.fst
fstrmepslocal
fstdeterminizestar --use-log=true
fstrmsymbols exp/tri3/graph/disambig_tid.int
fstminimizeencoded
fstisstochastic exp/tri3/graph/HCLGa.fst
0.000444886 -0.0175772
HCLGa is not stochastic
add-self-loops --self-loop-scale=0.1 --reorder=true exp/tri3/final.mdl
steps/decode_fmllr.sh --nj 5 --cmd run.pl --mem 4G exp/tri3/graph data/dev exp/tri3/decode_dev
steps/decode.sh --scoring-opts --num-threads 1 --skip-scoring false --acwt 0.083333 --nj 5 --cmd run.pl --mem 4G --beam 10.0 --model exp/tri3/final.alimdl --max-active 2000 exp/tri3/graph data/dev exp/tri3/decode_dev.si
decode.sh: feature type is lda
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri3/graph exp/tri3/decode_dev.si
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_dev.si/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(2,9,34) and mean=15.6
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_dev.si/log/analyze_lattice_depth_stats.log
steps/decode_fmllr.sh: feature type is lda
steps/decode_fmllr.sh: getting first-pass fMLLR transforms.
steps/decode_fmllr.sh: doing main lattice generation phase
steps/decode_fmllr.sh: estimating fMLLR transforms a second time.
steps/decode_fmllr.sh: doing a final pass of acoustic rescoring.
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri3/graph exp/tri3/decode_dev
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_dev/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(1,5,16) and mean=7.7
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_dev/log/analyze_lattice_depth_stats.log
steps/decode_fmllr.sh --nj 5 --cmd run.pl --mem 4G exp/tri3/graph data/test exp/tri3/decode_test
steps/decode.sh --scoring-opts --num-threads 1 --skip-scoring false --acwt 0.083333 --nj 5 --cmd run.pl --mem 4G --beam 10.0 --model exp/tri3/final.alimdl --max-active 2000 exp/tri3/graph data/test exp/tri3/decode_test.si
decode.sh: feature type is lda
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri3/graph exp/tri3/decode_test.si
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_test.si/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(2,10,37) and mean=16.8
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_test.si/log/analyze_lattice_depth_stats.log
steps/decode_fmllr.sh: feature type is lda
steps/decode_fmllr.sh: getting first-pass fMLLR transforms.
steps/decode_fmllr.sh: doing main lattice generation phase
steps/decode_fmllr.sh: estimating fMLLR transforms a second time.
steps/decode_fmllr.sh: doing a final pass of acoustic rescoring.
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/tri3/graph exp/tri3/decode_test
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_test/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(1,5,18) and mean=8.6
steps/diagnostic/analyze_lats.sh: see stats in exp/tri3/decode_test/log/analyze_lattice_depth_stats.log
============================================================================ SGMM2 Training & Decoding ============================================================================ steps/align_fmllr.sh --nj 30 --cmd run.pl --mem 4G data/train data/lang exp/tri3 exp/tri3_ali steps/align_fmllr.sh: feature type is lda steps/align_fmllr.sh: compiling training graphs steps/align_fmllr.sh: aligning data in data/train using exp/tri3/final.alimdl and speaker-independent features. steps/align_fmllr.sh: computing fMLLR transforms steps/align_fmllr.sh: doing final alignment. steps/align_fmllr.sh: done aligning data. steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 4G data/lang exp/tri3_ali steps/diagnostic/analyze_alignments.sh: see stats in exp/tri3_ali/log/analyze_alignments.log
steps/train_ubm.sh --cmd run.pl --mem 4G 400 data/train data/lang exp/tri3_ali exp/ubm4
steps/train_ubm.sh: feature type is lda
steps/train_ubm.sh: using transforms from exp/tri3_ali
steps/train_ubm.sh: clustering model exp/tri3_ali/final.mdl to get initial UBM
steps/train_ubm.sh: doing Gaussian selection
Pass 0
Pass 1
Pass 2
steps/train_sgmm2.sh --cmd run.pl --mem 4G 7000 9000 data/train data/lang exp/tri3_ali exp/ubm4/final.ubm exp/sgmm2_4
steps/train_sgmm2.sh: feature type is lda
steps/train_sgmm2.sh: using transforms from exp/tri3_ali
steps/train_sgmm2.sh: accumulating tree stats
steps/train_sgmm2.sh: Getting questions for tree clustering.
steps/train_sgmm2.sh: Building the tree
steps/train_sgmm2.sh: Initializing the model
steps/train_sgmm2.sh: doing Gaussian selection
steps/train_sgmm2.sh: compiling training graphs
steps/train_sgmm2.sh: converting alignments
steps/train_sgmm2.sh: training pass 0 ...
steps/train_sgmm2.sh: training pass 1 ...
steps/train_sgmm2.sh: training pass 2 ...
steps/train_sgmm2.sh: training pass 3 ...
steps/train_sgmm2.sh: training pass 4 ...
steps/train_sgmm2.sh: training pass 5 ...
steps/train_sgmm2.sh: re-aligning data
steps/train_sgmm2.sh: training pass 6 ...
steps/train_sgmm2.sh: training pass 7 ...
steps/train_sgmm2.sh: training pass 8 ...
steps/train_sgmm2.sh: training pass 9 ...
steps/train_sgmm2.sh: training pass 10 ...
steps/train_sgmm2.sh: re-aligning data
steps/train_sgmm2.sh: training pass 11 ...
steps/train_sgmm2.sh: training pass 12 ...
steps/train_sgmm2.sh: training pass 13 ...
steps/train_sgmm2.sh: training pass 14 ...
steps/train_sgmm2.sh: training pass 15 ...
steps/train_sgmm2.sh: re-aligning data
steps/train_sgmm2.sh: training pass 16 ...
steps/train_sgmm2.sh: training pass 17 ...
steps/train_sgmm2.sh: training pass 18 ...
steps/train_sgmm2.sh: training pass 19 ...
steps/train_sgmm2.sh: training pass 20 ...
steps/train_sgmm2.sh: training pass 21 ...
steps/train_sgmm2.sh: training pass 22 ...
steps/train_sgmm2.sh: training pass 23 ...
steps/train_sgmm2.sh: training pass 24 ...
steps/train_sgmm2.sh: building alignment model (pass 25)
steps/train_sgmm2.sh: building alignment model (pass 26)
steps/train_sgmm2.sh: building alignment model (pass 27)
198 warnings in exp/sgmm2_4/log/update_ali.*.log
1723 warnings in exp/sgmm2_4/log/update.*.log
1 warnings in exp/sgmm2_4/log/compile_questions.log
Done
tree-info exp/sgmm2_4/tree
tree-info exp/sgmm2_4/tree
make-h-transducer --disambig-syms-out=exp/sgmm2_4/graph/disambig_tid.int --transition-scale=1.0 data/lang_test_bg/tmp/ilabels_3_1 exp/sgmm2_4/tree exp/sgmm2_4/final.mdl
fsttablecompose exp/sgmm2_4/graph/Ha.fst data/lang_test_bg/tmp/CLG_3_1.fst
fstrmsymbols exp/sgmm2_4/graph/disambig_tid.int
fstrmepslocal
fstdeterminizestar --use-log=true
fstminimizeencoded
fstisstochastic exp/sgmm2_4/graph/HCLGa.fst
0.000485195 -0.0175772
HCLGa is not stochastic
add-self-loops --self-loop-scale=0.1 --reorder=true exp/sgmm2_4/final.mdl
steps/decode_sgmm2.sh --nj 5 --cmd run.pl --mem 4G --transform-dir exp/tri3/decode_dev exp/sgmm2_4/graph data/dev exp/sgmm2_4/decode_dev
steps/decode_sgmm2.sh: feature type is lda
steps/decode_sgmm2.sh: using transforms from exp/tri3/decode_dev
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/sgmm2_4/graph exp/sgmm2_4/decode_dev
steps/diagnostic/analyze_lats.sh: see stats in exp/sgmm2_4/decode_dev/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(2,6,20) and mean=9.5
steps/diagnostic/analyze_lats.sh: see stats in exp/sgmm2_4/decode_dev/log/analyze_lattice_depth_stats.log
steps/decode_sgmm2.sh --nj 5 --cmd run.pl --mem 4G --transform-dir exp/tri3/decode_test exp/sgmm2_4/graph data/test exp/sgmm2_4/decode_test
steps/decode_sgmm2.sh: feature type is lda
steps/decode_sgmm2.sh: using transforms from exp/tri3/decode_test
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 4G exp/sgmm2_4/graph exp/sgmm2_4/decode_test
steps/diagnostic/analyze_lats.sh: see stats in exp/sgmm2_4/decode_test/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(2,7,24) and mean=11.0
steps/diagnostic/analyze_lats.sh: see stats in exp/sgmm2_4/decode_test/log/analyze_lattice_depth_stats.log