专为数据科学和机器学习工作流程而设计,是一个开源包管理器,环境管理器,以及Python和R编程语言的分发。它通常用于大规模数据处理,科学计算和预测分析。
pip install xxx ,在特定环境里使用pip,下载的包会存在特定环境的目录里面
,例如:D:Anaconda3envs lpLibsite-packagesfasttext;conda install xxx ,不管在什么环境下载的包,都统一放在一个目录里面:D:Anaconda3pkgsfasttext,
如果另一个环境也需要这个包,不需要重新下载;
Anaconda:
conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/
conda config --set show_channel_urls yes
#-------
conda config --add channels conda-forge #添加conda 源
#查看所有的 conda .condarc 配置文件内容
conda config --get
- 要充分利用conda 包的好处,遇到需要安装什么包的优选conda:https://anaconda.org/wdroz/openpose-gpu
- 通过源码编译的是在其他已经都编译的没有的状况下,或者需要修改源码
- 设置不进入conda环境,新版的anaconda(python3.7)安装后自动进入虚拟环境base,执行命令:
conda config --set auto_activate_base false
conda package
-
A conda package is a compressed tarball file (.tar.bz2) or .conda file that contains:
- system-level libraries.
- Python or other modules.
- executable programs and other components.
- metadata under the
info/
directory. - a collection of files that are installed directly into an
install
prefix.
-
package structure:
├── bin #relevant binaries for package │ └── pyflakes ├── info # package metadata │ ├── LICENSE.txt │ ├── files │ ├── index.json │ ├── paths.json │ └── recipe └── lib # relevant library files └── python3.5
-
metadata contain dependencies to system core, low-level libraries and can contain links to software files that are automatically downloaded when executed.
-
index.json
- metadata about the package including platform, version, dependencies, and build info
{ "arch": "x86_64", "build": "py37hfa4b5c9_1", "build_number": 1, "depends": [ "depend > 1.1.1" ], "license": "BSD 3-Clause", "name": "fun-packge", "platform": "linux", "subdir": "linux-64", "timestamp": 1535416612069, "version": "0.0.0" }
paths.json
- a list of files in the package, along with their associated SHA-256, size in bytes, and the type of path (eg. hardlink vs. softlink)
{ "paths": [ { "_path": "lib/python3.7/site-packages/fun-packge/__init__.py", "path_type": "hardlink", "sha256": "76f3b6e34feeb651aff33ca59e0279c4eadce5a50c6ad93b961c846f7ba717e9", "size_in_bytes": 2067 }, { "_path": "lib/python3.7/site-packages/fun-packge/__config__.py", "path_type": "hardlink", "sha256": "348e3602616c1fe4c84502b1d8cf97c740d886002c78edab176759610d287f06", "size_in_bytes": 87519 }, ... }
-
Conda Dir
- root_dir:
pkgs
: contains decompressed packages, ready to be linked in conda environments. Each package resides in a subdirectory corresponding to its canonical name.envs
: conda environments- bin
- include
- share
.condarc
- Where conda
looks for packages
. - how conda uses a
proxy server.
- Where conda lists
known environments
. - Whether to update the Bash prompt with the currently activated environment name.
- Whether user-built packages should be uploaded to Anaconda.org.
- What default packages or features to include in new environments.
Conda 下载流程
Conda follows these steps when installing a package:
- Downloading and processing
index metadata
. - Reducing the index.
- Expressing the
package data
and constraints as a SAT problem. - Running the solver.
- Downloading and extracting packages.
- Verifying package contents.
- Linking packages from package cache into environments.
包首先现在到这个目录下
环境打包
# 1.创建环境的快照或备份
conda create --name snapshot --clone myenv
# 2. 复制环境 相同操作系统
# 生成 spec list 文件
conda list --explicit > spec-list.txt
pip freeze > requirements.txt
# 创建环境
conda create --name myname --file spec-list.txt
# 3. 生成environment.yml 跨平台操作系统复现环境
# 导出environment.yml 文件, 包括 pip 安装软件
conda env export > environment.yml
#创建环境
conda env create -f environment.yml
pip install -r requirements.txt
# 4.Pack 打包,包括该环境中安装的软件包的所有二进制文件, conda-pack 指定平台和操作系统,目标计算机必须集有与源计算机相同的平台和操作系统
# 安装
conda install -c conda-forge conda-pack
# 打包一个环境
# pack environment my_env into my_env.tar.gz
conda pack -n my_env
# pack environment my_env into out_name.tar.gz
conda pack -n my_env -o out_name.tar.gz
# pack environment located at an explicit path into my_env.tar.gz
conda pack -p /explicit/path/to/my_env
# 创建环境
mkdir -p my_env
tar -xzvf my_env.tar.gz -C my_env
# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
./my_env/bin/python
# Activate the environment. This adds `my_env/bin` to your path
source my_env/bin/activate
# Run Python from in the environment
(my_env) $ python
# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack
包管理
#方式一
# 文件夹那放到对应python 环境目录下lib->sitepackages然后import 就行。例如 winguiauto.py 文件放在Anaconda3envspytorch-openposeLibsite-packages, 直接import winguiauto ,可能需要重启下
#Tips python 目录(pytorch-openpose) >where python
#C:UsersdellAnaconda3envspytorch-openposepython.exe
#C:UsersdellAnaconda3python.exe #C:UsersdellAppDataLocalMicrosoftWindowsAppspython.exe
#方式二: 批量包下载
# 新建 .txt 文件 内容(h5requipy,numpy,tensorflow==2.0.0-alpha0)单独成行,
pip install -r requirement.txt #进行下载 -i https://pypi.tuna.tsinghua.edu.cn/simple/
#方式三: 直接 conda install / pip install 进行下载
#方式四: 离线下载
# 直接下载的whl文件,通过pip install (路径+whl文件名)
# 可以下载到本地 anacondainstallLibsite-packages路径下,或者在线下载安装
#查看版本
conda --version
conda update conda
#环境管理
conda create --name pysot python=3.7
pip install -r requirements.txt
conda info --envs #conda env list
conda activate pysot
conda deactivate pysot
#根据environment.yml 创建环境
conda env create -f environment.yml #yml 第一行是环境名称
conda remove --name ENVNAME --all
#环境迁移
#包管理
conda search package
conda install package
conda uninstall package --name env
conda install scipy=0.15.0 #下载指定版本
conda list # list package
conda list -n env #查看一个环境中所有的包
#删除包
conda remove -n myenv scipy
#install with channel
conda install scipy --channel conda-forge
#本地下载 本地下载不会解决包依赖问题
conda install /packages-path/packages-filename.tar
conda config --set env_prompt '({name})'
查看包依赖
- The tarball is unarchived in the pkgs directory
- Files are hard-linked to the install path
- Shebang lines and other instance of a place-holder prefix are replaced with the install prefix
- The metadata is updated, so that conda knows that it is installed
- the process of command conda create -n py3k python=3.7
- Environments are simple: just link the package to a different directory
- Hand-links are very cheap, and very fast
- Conda environments are completely independent installations of everything
- No fiddling with pythonPath or symlinking site-package
- Activating an environment just means changing your path so that its bin/ or scripts/ comes first
To find what packages are depending on a specific package in your environment
, there is not one specific conda command. It requires a series of steps:
- List the dependencies that a specific package requires to run:
conda info package_name
- Find your installation’s package cache directory:
conda info
- Find package dependencies. By default, Anaconda/Miniconda stores packages in ~/anaconda/pkgs/ (or ~/opt/pkgs/ on macOS Catalina). Each package has an index.json file which lists the package’s dependencies. This file resides in ~anaconda/pkgs/package_name/info/index.json.
- Now you can find what packages depend on a specific package. Use grep to search all index.json files as follows:
grep package_name ~/anaconda/pkgs/*/info/index.json
包导入原理:
-
怎么加载?——加载方式和用法
- import package 读这个包的__init__.py,也就是说导入包的本质是执行包下面的__init__.py文件,执行结束后会包名的目录下生成一个"pycache / init.cpython-36.pyc" 文件
- import module 读整个模块的内容
- import package1.package2.package3.module4 package读取__init__.py,module读取整个模块内容,按顺序读注意:因为package是读取__init__.py,所以调用的时候必须在__init__.py有引用的东西才能调用,否则会报错。
- 从指定路径导入:
import sys sys.path.append('/path/to/folder/containing/my_package') #注意是要导入包的上级目录 import my_package
-
案例
├── sub_ui.py├── main.py├── test/│ └── test1.py└── model/ └── model1.py#----- 同级目录导入 main.py导入sub_ui.pyimport sub_ui 或from sub_ui import *#----- 子目录,main.py导入model/model1.py需要在model/下建立__init__.py空文件,让编译器认为这是一个模块。 import model.model1或 from model.model1 import *#---- 跨目录导入 跨目录,model1.py导入test/test1.py需要在test1/下建立__init__.py空文件import sys sys.path.append("..") import test.test1 或者 import sys sys.path.append("..") import test.test1 import *
- 从哪里导入:python中,每个.py文件被称之为模块,每个具有
__init__.py
文件的目录被称为包。只要模块或者包所在的目录在sys.path
中,就可以使用import 模块或import 包来使用.对于python来说,所有被加载到内存的模块都是放在sys.modules里面,所以执行import时会首先去该列表中查询是否已添加。如果已经在sys.modules中,那就简单了,只需要将该module的name添加到我们正在调用该module的本地空间中。如果还没有放在sys.modules中,就需要在sys.path所有路径的目录中去按顺序查找该模块的文件,这些文件一般后缀为".py"、".pyo"、".pyc"、".pyd"、".dll",找到这些模块后就可以将这些模块添加到sys.modules中,再将module name导入到本地。
>>> import sys>>> sys.modules{'copy_reg': <module 'copy_reg' from '/usr/lib/python2.7/copy_reg.pyc'>, 'sre_compile': <module 'sre_compile' from '/usr/lib/python2.7/sre_compile.pyc'>, '_sre': <module '_sre' (built-in)>, 'encodings': <module 'encodings' from '/usr/lib/python2.7/encodings/__init__.pyc'>, 'site': <module 'site' from '/usr/lib/python2.7/site.pyc'>, '__builtin__': <module '__builtin__' (built-in)>, 'sysconfig': <module 'sysconfig' from '/usr/lib/python2.7/sysconfig.pyc'>, '__main__': <module '__main__' (built-in)>, 'encodings.encodings': None, 'abc': <module 'abc' from '/usr/lib/python2.7/abc.pyc'>, 'posixpath': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_weakrefset': <module '_weakrefset' from '/usr/lib/python2.7/_weakrefset.pyc'>, 'errno': <module 'errno' (built-in)>, 'encodings.codecs': None, 'sre_constants': <module 'sre_constants' from '/usr/lib/python2.7/sre_constants.pyc'>, 're': <module 're' from '/usr/lib/python2.7/re.pyc'>, '_abcoll': <module '_abcoll' from '/usr/lib/python2.7/_abcoll.pyc'>, 'types': <module 'types' from '/usr/lib/python2.7/types.pyc'>, '_codecs': <module '_codecs' (built-in)>, 'encodings.__builtin__': None, '_warnings': <module '_warnings' (built-in)>, 'genericpath': <module 'genericpath' from '/usr/lib/python2.7/genericpath.pyc'>, 'stat': <module 'stat' from '/usr/lib/python2.7/stat.pyc'>, 'zipimport': <module 'zipimport' (built-in)>, '_sysconfigdata': <module '_sysconfigdata' from '/usr/lib/python2.7/_sysconfigdata.pyc'>, 'mpl_toolkits': <module 'mpl_toolkits' (built-in)>, 'warnings': <module 'warnings' from '/usr/lib/python2.7/warnings.pyc'>, 'UserDict': <module 'UserDict' from '/usr/lib/python2.7/UserDict.pyc'>, 'encodings.utf_8': <module 'encodings.utf_8' from '/usr/lib/python2.7/encodings/utf_8.pyc'>, 'sys': <module 'sys' (built-in)>, 'codecs': <module 'codecs' from '/usr/lib/python2.7/codecs.pyc'>, 'readline': <module 'readline' from '/usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so'>, '_sysconfigdata_nd': <module '_sysconfigdata_nd' from '/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc'>, 'os.path': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_locale': <module '_locale' (built-in)>, 'sitecustomize': <module 'sitecustomize' from '/usr/lib/python2.7/sitecustomize.pyc'>, 'signal': <module 'signal' (built-in)>, 'traceback': <module 'traceback' from '/usr/lib/python2.7/traceback.pyc'>, 'linecache': <module 'linecache' from '/usr/lib/python2.7/linecache.pyc'>, 'posix': <module 'posix' (built-in)>, 'encodings.aliases': <module 'encodings.aliases' from '/usr/lib/python2.7/encodings/aliases.pyc'>, 'exceptions': <module 'exceptions' (built-in)>, 'sre_parse': <module 'sre_parse' from '/usr/lib/python2.7/sre_parse.pyc'>, 'os': <module 'os' from '/usr/lib/python2.7/os.pyc'>, '_weakref': <module '_weakref' (built-in)>}>>> sys.path['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/ldd/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']
Error record:
- Missing dependencies for SOCKS support. 解决方法: unset all_proxy
linux 安装Anaconda
#本地安装pip install ~/Dowloads/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl#conda 安装本地包conda install --use-local pytorch-0.4.0-py35_cuda8.0.61_cudnn7.1.2_1.tar.bz2#change the channelconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --set show_channel_urls yes#下载安装curl -O https://repo.anaconda.com/archive/Anaconda3-5.1.0-Linux-x86_64.sh#通过SHA-256校验和通过加密哈希验证来验证安装程序的数据完整性sha256sum Anaconda3-5.1.0-Linux-x86_64.sh#安装bash Anaconda3-5.1.0-Linux-x86_64.sh#激活环境,否则无法找到conda命令source ~/.bashrcconda info#用来查看conda 内部安装环境版本等信息。#验证安装是否正确 罗列conda 安装所有软件conda list#创建 一个环境my_env Python版本为3conda create --name my_env python=3 #使用 sudo 命令创建,否则无法在环境中注册#在环境my_env35中安装numpy包 conda install --name my_env35 numpy#激活环境 ,这样会有个前缀#更新 conda使用程序conda update conda#更新 或卸载 anaconda 安装包conda remove -n envname --all #delete all the package and environmentconda update anaconda conda install anaconda-cleanrm -rf anacondasudo gedit ~/.bashrc #删除环境变量source ~/.bashrc #使环境变量生效
Pycharm :
#PyCharm:https://www.jetbrains.com/pycharm/download/#section=linuxcd downloads/pycharm-2018.1/binbash pycharm.sh#进入file setting 设置选择编译器
Win环境配置
F:Anaconda3F:Anaconda3ScriptsF:Anaconda3Libraryin#shell 中加入这三个目录,才可以在cmd 中使用conda
Jupyter使用
若想要关闭已经打开的终端和“ipynb”格式的笔记本,仅仅关闭其页面是无法彻底退出程序的,需要在Running页面点击其对应的“Shutdown”。
-
安装: conda install jupyter notebook
-
修改默认的主目录:jupyter notebook --generate-config
-
启动目录: jupyter notebook, http://localhost:8888; 可以对端口进行映射处理
- 指定端口号来启动: jupyter notebook --port <port_number>
- 启动服务但不打开网页: jupyter notebook --no-browser
- 在conda 虚拟环境中运行启动 jupyter 命令,kernel 会自动关联所有的conda环境;
-
帮助命令: jupyter notebook --help
-
将你conda创建的环境与Jupyter Notebook相关联: conda install nb_conda
-
加载本地python文件:%load Python文件的绝对路径
-
直接运行本地python文件:%run Python文件的绝对路径; !python3 Python文件的绝对路径; !shell命令;
-
ModuleNotFoundError: No module named 'jupyter_nbextensions_configurator'
-
pip install jupyter_nbextensions_configurator
-
-
Error executing Jupyter command 'notebook': [Errno 2] No such file or directory
- jupyter notebook --port=11220 --allow-root
#不适用sudo 权限就可以执行了
- jupyter notebook --port=11220 --allow-root
-
Jupyter 常用快捷键
ctrl + /
#进行注释cell操作
Ctrl+enter 执行本cell
shift+enter 执行本cell且 向下建立一个新cell
A 向上建立一个cell
B 向下建立一个cell
d d 删除cell
O 收起output 或者打开output
在cell里面的快捷键
Esc+m m 把cell切换至markdown模式
Esc+y y 把cell切换至code模式
Esc+l l 显示行数
Esc + F 在代码中查找、替换,忽略输出。
Esc + O 在 cell 和输出结果间切换。复合快捷键
shift 选中多个cell
然后执行 shift+M 合并选中cell
Error Record
.1. 代理问题
#Anaconda 使用conda指令出现ValueError: Unable to determine SOCKS version from socks://127.0.0.1:1080/ 解决办法printenv | grep all_proxy #查看代理unset all_proxy && unset ALL_PROXY #export all_proxy="" export all_proxy="socks5://127.0.0.1:1080"export http_proxyexport https_proxyunset http_proxyunset https_proxyconda install pysocks #下次创建环境是同时下载 pip和pysocks 包source ~/.bashrc
每次在虚拟环境内部使用pip
的时候都需要获取虚拟环境内部的 pip
命令路径:/Users/stefanannihilater/anaconda3/envs/setests3/bin/pip
,显然这样非常的繁琐。
如果想在虚拟环境内部使用简单的pip install package
调用虚拟环境内部的pip
命令的话,只需要我们在创建虚拟环境的时候指定pip
只对虚拟环境生效,而不影响全局库:
conda create -n 虚拟环境名 pip pysocks
pip
:是为了指定pip
命令只对当前虚拟环境生效pysocks
:是pip
命令的依赖,如果不写,在虚拟环境内使用pip
命令的时候会出现Missing dependencies for SOCKS support.
的报错。
下次可以把包下载到本地,然后局域网上传服务器,减少服务器流量开销;
-
错误记录SSL
- 方式一:环境变量是否配好:
D:Anaconda3; D:Anaconda3Scripts; D:Anaconda3Libraryin
- 方式二: 镜像网址是否可以访问
- 方式三: 安装openssl 工具:地址
.2. cudatoolkit=11.1. 下载失败
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/conda config --set show_channel_urls yesconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/conda install pytorch torchvision torchaudio cudatoolkit=11.1
.3. torchvision 库无法使用
Download error with CIFAR10/Mnist based on torchvision.datasets
- uninstall and reinstall torchvision
conda unstall torchvisionconda install pytorch torchvision torchaudio cudatoolkit=11.1
.4. pip 代理
.5. GPU资源问题
RuntimeError:
cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
;
- GPU被沾满了,没有分配资源;