• IDL double与float的精度检验


    pro test
      compile_opt idl2
      a = 100.000000002
      b = 100.000000000
      help, a
      help, b
      if a gt b then $
        print, '100.000000002 gt 100.000000000'
      if a eq b then $
        print, '100.000000002 eq 100.000000000'
      if a lt b then $
        print, '100.000000002 lt 100.000000000'
      a = 100.000000002d
      b = 100.000000000d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002d gt 100.000000000d'
      if a eq b then $
        print, '100.000000002d eq 100.000000000d'
      if a lt b then $
        print, '100.000000002d lt 100.000000000d'
      print, ''
      a = double(100.000000002)
      b = double(100.000000000)
      help, a
      help, b
      if a gt b then $
        print, 'double(100.000000002) gt double(100.000000000)'
      if a eq b then $
        print, 'double(100.000000002) eq double(100.000000000)'
      if a lt b then $
        print, 'double(100.000000002) lt double(100.000000000)'
      print, ''
      a = 100.000000002*1.0d
      b = 100.000000000*1.0d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002*1.0d gt 100.000000000*1.0d'
      if a eq b then $
        print, '100.000000002*1.0d eq 100.000000000*1.0d'
      if a lt b then $
        print, '100.000000002*1.0d lt 100.000000000*1.0d'
      print, ''
     
      a = double('100.1234567892')
      b = double('100.1234567890')
      help, a
      help, b
      if a gt b then $
        print, '100.1234567892*1.0d gt 100.1234567890*1.0d'
      if a eq b then $
        print, '100.1234567892*1.0d eq 100.1234567890*1.0d'
      if a lt b then $
        print, '100.1234567892*1.0d lt 100.1234567890*1.0d'
      print, ''
    end
     
    运行结果
    A               FLOAT     =       100.000
    B               FLOAT     =       100.000
    100.000000002 eq 100.000000000
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002d gt 100.000000000d
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    double(100.000000002) eq double(100.000000000)
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002*1.0d eq 100.000000000*1.0d
     
    A               DOUBLE    =        100.12346
    B               DOUBLE    =        100.12346
    str100.1234567892 gt str100.1234567890
     
    初始的数据如果不是double型的,后面转为double型是没有用的,因为精度已经丢失
    从a=100.000000002 可知,虽然是小数点后9位, 但IDL认为是float型
    a=100.000000002d 可知,加上d之后,IDL认为是double型
    从a = double('100.1234567892') 可知,读入一个位数很多的字符串,将其转为double,可以最大限度的保持后续计算的精度
     
     
     
  • 相关阅读:
    Visual Studio Code 配置C/C++环境
    二叉链的基本操作
    回文自动机
    吊打线段树的超级树状数组
    Treap平衡树
    uni-app nvue页面动态修改导航栏按钮
    uni-app map组件关于marker标记点动态设置的问题
    uni-app 提示 v-for 暂不支持循环数据
    uni-app APP端隐藏导航栏自定义按钮
    uni-app路径规划(打开第三方地图实现)
  • 原文地址:https://www.cnblogs.com/lqqgis/p/13626557.html
Copyright © 2020-2023  润新知