• 【Tools】Anaconda Operaction


    专为数据科学和机器学习工作流程而设计,是一个开源包管理器,环境管理器,以及Python和R编程语言的分发。它通常用于大规模数据处理,科学计算和预测分析。pip install xxx ,在特定环境里使用pip,下载的包会存在特定环境的目录里面,例如:D:Anaconda3envs lpLibsite-packagesfasttext; conda install xxx ,不管在什么环境下载的包,都统一放在一个目录里面:D:Anaconda3pkgsfasttext, 如果另一个环境也需要这个包,不需要重新下载;

    Anaconda:

    conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
    conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
    conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/
    conda config --set show_channel_urls yes
    #-------
    conda config --add channels conda-forge   #添加conda 源
    #查看所有的 conda .condarc 配置文件内容
    conda config --get
    
    • 要充分利用conda 包的好处,遇到需要安装什么包的优选conda:https://anaconda.org/wdroz/openpose-gpu
    • 通过源码编译的是在其他已经都编译的没有的状况下,或者需要修改源码
    • 设置不进入conda环境,新版的anaconda(python3.7)安装后自动进入虚拟环境base,执行命令:

    ​ conda config --set auto_activate_base false

    conda package

    • A conda package is a compressed tarball file (.tar.bz2) or .conda file that contains:

      • system-level libraries.
      • Python or other modules.
      • executable programs and other components.
      • metadata under the info/ directory.
      • a collection of files that are installed directly into an install prefix.
    • package structure:

      ├── bin            #relevant binaries for package
      │   └── pyflakes
      ├── info     # package metadata
      │   ├── LICENSE.txt
      │   ├── files
      │   ├── index.json
      │   ├── paths.json
      │   └── recipe
      └── lib       # relevant library files
          └── python3.5
      
      • metadata contain dependencies to system core, low-level libraries and can contain links to software files that are automatically downloaded when executed.

      • index.json

        • metadata about the package including platform, version, dependencies, and build info
      {
        "arch": "x86_64",
        "build": "py37hfa4b5c9_1",
        "build_number": 1,
        "depends": [
          "depend > 1.1.1"
        ],
        "license": "BSD 3-Clause",
        "name": "fun-packge",
        "platform": "linux",
        "subdir": "linux-64",
        "timestamp": 1535416612069,
        "version": "0.0.0"
      }
      
      • paths.json
        • a list of files in the package, along with their associated SHA-256, size in bytes, and the type of path (eg. hardlink vs. softlink)
      {
        "paths": [
          {
            "_path": "lib/python3.7/site-packages/fun-packge/__init__.py",
            "path_type": "hardlink",
            "sha256": "76f3b6e34feeb651aff33ca59e0279c4eadce5a50c6ad93b961c846f7ba717e9",
            "size_in_bytes": 2067
          },
          {
            "_path": "lib/python3.7/site-packages/fun-packge/__config__.py",
            "path_type": "hardlink",
            "sha256": "348e3602616c1fe4c84502b1d8cf97c740d886002c78edab176759610d287f06",
            "size_in_bytes": 87519
          },
          ...
      }
      

    Conda Dir

    • root_dir:
    • pkgs: contains decompressed packages, ready to be linked in conda environments. Each package resides in a subdirectory corresponding to its canonical name.
    • envs: conda environments
      • bin
      • include
      • share
    .condarc
    • Where conda looks for packages.
    • how conda uses a proxy server.
    • Where conda lists known environments.
    • Whether to update the Bash prompt with the currently activated environment name.
    • Whether user-built packages should be uploaded to Anaconda.org.
    • What default packages or features to include in new environments.
    Conda 下载流程
    ../../_images/installing-with-conda.png

    Conda follows these steps when installing a package:

    1. Downloading and processing index metadata.
    2. Reducing the index.
    3. Expressing the package data and constraints as a SAT problem.
    4. Running the solver.
    5. Downloading and extracting packages.
    6. Verifying package contents.
    7. Linking packages from package cache into environments.

    包首先现在到这个目录下

    环境打包

    #  1.创建环境的快照或备份
    conda create --name snapshot --clone myenv
    
    #  2. 复制环境  相同操作系统
    # 生成 spec list 文件
    conda list --explicit > spec-list.txt
    pip freeze > requirements.txt
    # 创建环境
    conda create --name myname --file spec-list.txt
    
    #  3. 生成environment.yml 跨平台操作系统复现环境
    # 导出environment.yml 文件, 包括 pip 安装软件
    conda env export > environment.yml
    #创建环境
    conda env create -f environment.yml
    pip install -r requirements.txt
    
    #  4.Pack 打包,包括该环境中安装的软件包的所有二进制文件, conda-pack 指定平台和操作系统,目标计算机必须集有与源计算机相同的平台和操作系统
    # 安装
    conda install -c conda-forge conda-pack
    # 打包一个环境
    # pack environment my_env into my_env.tar.gz
    conda pack -n my_env
    # pack environment my_env into out_name.tar.gz
    conda pack -n my_env -o out_name.tar.gz
    # pack environment located at an explicit path into my_env.tar.gz
    conda pack -p /explicit/path/to/my_env
    # 创建环境
    mkdir -p my_env
    tar -xzvf my_env.tar.gz -C my_env
    # Use Python without activating or fixing the prefixes. Most Python
    # libraries will work fine, but things that require prefix cleanups
    # will fail.
    ./my_env/bin/python
    
    # Activate the environment. This adds `my_env/bin` to your path
    source my_env/bin/activate
    
    # Run Python from in the environment
    (my_env) $ python
    
    # Cleanup prefixes from in the active environment.
    # Note that this command can also be run without activating the environment
    # as long as some version of Python is already installed on the machine.
    (my_env) $ conda-unpack
    

    包管理

    #方式一
    #	文件夹那放到对应python 环境目录下lib->sitepackages然后import 就行。例如 winguiauto.py 文件放在Anaconda3envspytorch-openposeLibsite-packages, 直接import winguiauto ,可能需要重启下
    #Tips  python 目录(pytorch-openpose) >where python
    	#C:UsersdellAnaconda3envspytorch-openposepython.exe
    	#C:UsersdellAnaconda3python.exe	#C:UsersdellAppDataLocalMicrosoftWindowsAppspython.exe    
    #方式二: 批量包下载 
    # 新建 .txt 文件 内容(h5requipy,numpy,tensorflow==2.0.0-alpha0)单独成行,
    pip install -r requirement.txt #进行下载  -i https://pypi.tuna.tsinghua.edu.cn/simple/
    
    #方式三: 直接 conda install / pip install 进行下载
    
    #方式四: 离线下载
    #	直接下载的whl文件,通过pip install (路径+whl文件名)
    #	可以下载到本地 anacondainstallLibsite-packages路径下,或者在线下载安装
    
    #查看版本
    conda --version
    conda update conda
    #环境管理
    conda create --name pysot python=3.7
    pip install -r requirements.txt
    conda info --envs        #conda env list
    conda activate pysot
    conda deactivate pysot
    #根据environment.yml 创建环境
    conda env create -f environment.yml  #yml 第一行是环境名称
    conda remove --name ENVNAME --all
    #环境迁移
    #包管理
    conda search package
    conda install package
    conda uninstall package --name env
    conda install scipy=0.15.0  #下载指定版本
    conda list  # list package
    conda list -n env  #查看一个环境中所有的包
    #删除包
    conda remove -n myenv scipy
    #install with channel
    conda install scipy --channel conda-forge
    #本地下载   本地下载不会解决包依赖问题
    conda install /packages-path/packages-filename.tar
    conda config --set env_prompt '({name})'
    

    查看包依赖

    • The tarball is unarchived in the pkgs directory
    • Files are hard-linked to the install path
    • Shebang lines and other instance of a place-holder prefix are replaced with the install prefix
    • The metadata is updated, so that conda knows that it is installed
    • the process of command conda create -n py3k python=3.7
    • Environments are simple: just link the package to a different directory
    • Hand-links are very cheap, and very fast
    • Conda environments are completely independent installations of everything
      • No fiddling with pythonPath or symlinking site-package
    • Activating an environment just means changing your path so that its bin/ or scripts/ comes first

    To find what packages are depending on a specific package in your environment, there is not one specific conda command. It requires a series of steps:

    1. List the dependencies that a specific package requires to run: conda info package_name
    2. Find your installation’s package cache directory: conda info
    3. Find package dependencies. By default, Anaconda/Miniconda stores packages in ~/anaconda/pkgs/ (or ~/opt/pkgs/ on macOS Catalina). Each package has an index.json file which lists the package’s dependencies. This file resides in ~anaconda/pkgs/package_name/info/index.json.
    4. Now you can find what packages depend on a specific package. Use grep to search all index.json files as follows: grep package_name ~/anaconda/pkgs/*/info/index.json

    包导入原理:

    • 怎么加载?——加载方式和用法

      1. import package 读这个包的__init__.py,也就是说导入包的本质是执行包下面的__init__.py文件,执行结束后会包名的目录下生成一个"pycache / init.cpython-36.pyc" 文件
      2. import module 读整个模块的内容
      3. import package1.package2.package3.module4 package读取__init__.py,module读取整个模块内容,按顺序读注意:因为package是读取__init__.py,所以调用的时候必须在__init__.py有引用的东西才能调用,否则会报错。
      4. 从指定路径导入:
      import sys
      sys.path.append('/path/to/folder/containing/my_package')  #注意是要导入包的上级目录
      import my_package	
      
    • 案例

    ├── sub_ui.py├── main.py├── test/│   └── test1.py└── model/    └── model1.py#-----  同级目录导入 main.py导入sub_ui.pyimport sub_ui  或from sub_ui import *#-----  子目录,main.py导入model/model1.py需要在model/下建立__init__.py空文件,让编译器认为这是一个模块。    import model.model1或    from model.model1 import *#----   跨目录导入 跨目录,model1.py导入test/test1.py需要在test1/下建立__init__.py空文件import sys sys.path.append("..") import test.test1 或者 import sys sys.path.append("..") import test.test1 import *
    
    •  从哪里导入:python中,每个.py文件被称之为模块,每个具有__init__.py文件的目录被称为包。只要模块或者包所在的目录在sys.path中,就可以使用import 模块或import 包来使用.对于python来说,所有被加载到内存的模块都是放在sys.modules里面,所以执行import时会首先去该列表中查询是否已添加。如果已经在sys.modules中,那就简单了,只需要将该module的name添加到我们正在调用该module的本地空间中。如果还没有放在sys.modules中,就需要在sys.path所有路径的目录中去按顺序查找该模块的文件,这些文件一般后缀为".py"、".pyo"、".pyc"、".pyd"、".dll",找到这些模块后就可以将这些模块添加到sys.modules中,再将module name导入到本地。
    >>> import sys>>> sys.modules{'copy_reg': <module 'copy_reg' from '/usr/lib/python2.7/copy_reg.pyc'>, 'sre_compile': <module 'sre_compile' from '/usr/lib/python2.7/sre_compile.pyc'>, '_sre': <module '_sre' (built-in)>, 'encodings': <module 'encodings' from '/usr/lib/python2.7/encodings/__init__.pyc'>, 'site': <module 'site' from '/usr/lib/python2.7/site.pyc'>, '__builtin__': <module '__builtin__' (built-in)>, 'sysconfig': <module 'sysconfig' from '/usr/lib/python2.7/sysconfig.pyc'>, '__main__': <module '__main__' (built-in)>, 'encodings.encodings': None, 'abc': <module 'abc' from '/usr/lib/python2.7/abc.pyc'>, 'posixpath': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_weakrefset': <module '_weakrefset' from '/usr/lib/python2.7/_weakrefset.pyc'>, 'errno': <module 'errno' (built-in)>, 'encodings.codecs': None, 'sre_constants': <module 'sre_constants' from '/usr/lib/python2.7/sre_constants.pyc'>, 're': <module 're' from '/usr/lib/python2.7/re.pyc'>, '_abcoll': <module '_abcoll' from '/usr/lib/python2.7/_abcoll.pyc'>, 'types': <module 'types' from '/usr/lib/python2.7/types.pyc'>, '_codecs': <module '_codecs' (built-in)>, 'encodings.__builtin__': None, '_warnings': <module '_warnings' (built-in)>, 'genericpath': <module 'genericpath' from '/usr/lib/python2.7/genericpath.pyc'>, 'stat': <module 'stat' from '/usr/lib/python2.7/stat.pyc'>, 'zipimport': <module 'zipimport' (built-in)>, '_sysconfigdata': <module '_sysconfigdata' from '/usr/lib/python2.7/_sysconfigdata.pyc'>, 'mpl_toolkits': <module 'mpl_toolkits' (built-in)>, 'warnings': <module 'warnings' from '/usr/lib/python2.7/warnings.pyc'>, 'UserDict': <module 'UserDict' from '/usr/lib/python2.7/UserDict.pyc'>, 'encodings.utf_8': <module 'encodings.utf_8' from '/usr/lib/python2.7/encodings/utf_8.pyc'>, 'sys': <module 'sys' (built-in)>, 'codecs': <module 'codecs' from '/usr/lib/python2.7/codecs.pyc'>, 'readline': <module 'readline' from '/usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so'>, '_sysconfigdata_nd': <module '_sysconfigdata_nd' from '/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc'>, 'os.path': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_locale': <module '_locale' (built-in)>, 'sitecustomize': <module 'sitecustomize' from '/usr/lib/python2.7/sitecustomize.pyc'>, 'signal': <module 'signal' (built-in)>, 'traceback': <module 'traceback' from '/usr/lib/python2.7/traceback.pyc'>, 'linecache': <module 'linecache' from '/usr/lib/python2.7/linecache.pyc'>, 'posix': <module 'posix' (built-in)>, 'encodings.aliases': <module 'encodings.aliases' from '/usr/lib/python2.7/encodings/aliases.pyc'>, 'exceptions': <module 'exceptions' (built-in)>, 'sre_parse': <module 'sre_parse' from '/usr/lib/python2.7/sre_parse.pyc'>, 'os': <module 'os' from '/usr/lib/python2.7/os.pyc'>, '_weakref': <module '_weakref' (built-in)>}>>> sys.path['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/ldd/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']
    

    Error record:

    • Missing dependencies for SOCKS support. 解决方法: unset all_proxy

    linux 安装Anaconda


    #本地安装pip install ~/Dowloads/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl#conda 安装本地包conda install --use-local pytorch-0.4.0-py35_cuda8.0.61_cudnn7.1.2_1.tar.bz2#change the channelconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --set show_channel_urls yes#下载安装curl -O https://repo.anaconda.com/archive/Anaconda3-5.1.0-Linux-x86_64.sh#通过SHA-256校验和通过加密哈希验证来验证安装程序的数据完整性sha256sum Anaconda3-5.1.0-Linux-x86_64.sh#安装bash Anaconda3-5.1.0-Linux-x86_64.sh#激活环境,否则无法找到conda命令source ~/.bashrcconda info#用来查看conda 内部安装环境版本等信息。#验证安装是否正确 罗列conda 安装所有软件conda list#创建 一个环境my_env  Python版本为3conda create --name my_env python=3   #使用 sudo 命令创建,否则无法在环境中注册#在环境my_env35中安装numpy包 conda install --name my_env35 numpy#激活环境 ,这样会有个前缀#更新 conda使用程序conda update conda#更新 或卸载 anaconda 安装包conda remove -n envname --all   #delete all the package and environmentconda update anaconda conda install anaconda-cleanrm -rf anacondasudo gedit ~/.bashrc #删除环境变量source ~/.bashrc #使环境变量生效
    

    Pycharm :

    #PyCharm:https://www.jetbrains.com/pycharm/download/#section=linuxcd downloads/pycharm-2018.1/binbash pycharm.sh#进入file setting 设置选择编译器
    

    Win环境配置

    F:Anaconda3F:Anaconda3ScriptsF:Anaconda3Libraryin#shell 中加入这三个目录,才可以在cmd 中使用conda
    

    Jupyter使用

    若想要关闭已经打开的终端和“ipynb”格式的笔记本,仅仅关闭其页面是无法彻底退出程序的,需要在Running页面点击其对应的“Shutdown”。

    • 安装: conda install jupyter notebook

    • 修改默认的主目录:jupyter notebook --generate-config

    • 启动目录: jupyter notebook, http://localhost:8888; 可以对端口进行映射处理

      • 指定端口号来启动: jupyter notebook --port <port_number>
      • 启动服务但不打开网页: jupyter notebook --no-browser
      • 在conda 虚拟环境中运行启动 jupyter 命令,kernel 会自动关联所有的conda环境;
    • 帮助命令: jupyter notebook --help

    • 将你conda创建的环境与Jupyter Notebook相关联: conda install nb_conda

    • 加载本地python文件:%load Python文件的绝对路径

    • 直接运行本地python文件:%run Python文件的绝对路径; !python3 Python文件的绝对路径; !shell命令;

    • 学习链接: https://www.jianshu.com/p/91365f343585#conda

    • ModuleNotFoundError: No module named 'jupyter_nbextensions_configurator'

      • pip install jupyter_nbextensions_configurator
        
    • Error executing Jupyter command 'notebook': [Errno 2] No such file or directory

      • jupyter notebook --port=11220 --allow-root #不适用sudo 权限就可以执行了
    • Jupyter 常用快捷键

      ctrl + / #进行注释

      cell操作
      Ctrl+enter 执行本cell
      shift+enter 执行本cell且 向下建立一个新cell

      A 向上建立一个cell
      B 向下建立一个cell
      d d 删除cell
      O 收起output 或者打开output

      在cell里面的快捷键
      Esc+m m 把cell切换至markdown模式
      Esc+y y 把cell切换至code模式
      Esc+l l 显示行数
      Esc + F 在代码中查找、替换,忽略输出。
      Esc + O 在 cell 和输出结果间切换。

      复合快捷键
      shift 选中多个cell
      然后执行 shift+M 合并选中cell

    Error Record

    .1. 代理问题

    #Anaconda 使用conda指令出现ValueError: Unable to determine SOCKS version from socks://127.0.0.1:1080/ 解决办法printenv | grep all_proxy  #查看代理unset all_proxy && unset ALL_PROXY   #export all_proxy=""  export all_proxy="socks5://127.0.0.1:1080"export http_proxyexport https_proxyunset http_proxyunset https_proxyconda install pysocks     #下次创建环境是同时下载 pip和pysocks 包source ~/.bashrc   
    

    每次在虚拟环境内部使用pip的时候都需要获取虚拟环境内部的 pip命令路径:/Users/stefanannihilater/anaconda3/envs/setests3/bin/pip,显然这样非常的繁琐。

    如果想在虚拟环境内部使用简单的pip install package调用虚拟环境内部的pip命令的话,只需要我们在创建虚拟环境的时候指定pip只对虚拟环境生效,而不影响全局库:

    conda create -n 虚拟环境名 pip pysocks
    
    • pip:是为了指定pip命令只对当前虚拟环境生效
    • pysocks:是pip命令的依赖,如果不写,在虚拟环境内使用 pip命令的时候会出现Missing dependencies for SOCKS support.的报错。

    下次可以把包下载到本地,然后局域网上传服务器,减少服务器流量开销;

    • 错误记录SSL

      • 方式一:环境变量是否配好:
      D:Anaconda3; D:Anaconda3Scripts; D:Anaconda3Libraryin 
      

    .2. cudatoolkit=11.1. 下载失败

    conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/conda config --set show_channel_urls yesconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/conda install pytorch torchvision torchaudio cudatoolkit=11.1
    

    .3. torchvision 库无法使用

    Download error with CIFAR10/Mnist based on torchvision.datasets

    • uninstall and reinstall torchvision
    conda unstall torchvisionconda install pytorch torchvision torchaudio cudatoolkit=11.1
    

    .4. pip 代理

    .5. GPU资源问题

    RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

    • GPU被沾满了,没有分配资源;
  • 相关阅读:
    第13组 Beta (2/6)(俞子君)
    第13组 Beta (1/6)(俞子君)
    第13组 Alpha (6/6)(俞子君)
    第13组 Alpha (5/6)(俞子君)
    第13组 Alpha (4/6)(俞子君)
    第13组 Alpha (3/6)(俞子君)
    第13组 Alpha (2/6)(俞子君)
    番茄蛋花汤口罩识别系统汇报
    β(56/6)
    β(34/6)
  • 原文地址:https://www.cnblogs.com/liu-dongdong/p/15213199.html
Copyright © 2020-2023  润新知