バイオインフォマティクス武者修行 #3 Tensorflow

tensorflowを使ってGPUによる解析をする場合

とりあえずまとめました。遺伝研スパコンでGPUの解析がしたい人は参考までに。

何も対策せずに起動すると以下のようなエラーが生じる。

2022-11-02 14:08:09.957352: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-02 14:08:13.686095: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/iceplant4561/local/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib64/R/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib64/R/lib:/home/iceplant4561/local/lib64:/opt/pkg/gcc/9.2.0/lib:/opt/pkg/gcc/9.2.0/lib64:/lustre7/software/intel/oneapi/vpl/2022.1.0/lib:/lustre7/software/intel/oneapi/tbb/2021.6.0/env/../lib/intel64/gcc4.8:/lustre7/software/intel/oneapi/mpi/2021.6.0//libfabric/lib:/lustre7/software/intel/oneapi/mpi/2021.6.0//lib/release:/lustre7/software/intel/oneapi/mpi/2021.6.0//lib:/lustre7/software/intel/oneapi/mkl/2022.1.0/lib/intel64:/lustre7/software/intel/oneapi/itac/2021.6.0/slib:/lustre7/software/intel/oneapi/ipp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/ippcp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/ipp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/dnnl/2022.1.0/cpu_dpcpp_gpu_dpcpp/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/gdb/intel64/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/libipt/intel64/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/dep/lib:/lustre7/software/intel/oneapi/dal/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib/x64:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib/oclfpga/host/linux64/lib:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/compiler/lib/intel64_lin:/lustre7/software/intel/oneapi/ccl/2021.6.0/lib/cpu_gpu_dpcpp:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/lib/release:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/ipp/lib/intel64:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/tbb/lib/intel64/gcc4.7:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/tbb/lib/intel64/gcc4.7:/opt/pkg/intel/parallel_studio_xe_2019/update3/debugger_2019/libipt/intel64/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/daal/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/home/iceplant4561/anaconda3/lib/
2022-11-02 14:08:13.686170: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-11-02 14:08:14.048927: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2022-11-02 14:09:10.844763: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/iceplant4561/local/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib64/R/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib64/R/lib:/home/iceplant4561/local/lib64:/opt/pkg/gcc/9.2.0/lib:/opt/pkg/gcc/9.2.0/lib64:/lustre7/software/intel/oneapi/vpl/2022.1.0/lib:/lustre7/software/intel/oneapi/tbb/2021.6.0/env/../lib/intel64/gcc4.8:/lustre7/software/intel/oneapi/mpi/2021.6.0//libfabric/lib:/lustre7/software/intel/oneapi/mpi/2021.6.0//lib/release:/lustre7/software/intel/oneapi/mpi/2021.6.0//lib:/lustre7/software/intel/oneapi/mkl/2022.1.0/lib/intel64:/lustre7/software/intel/oneapi/itac/2021.6.0/slib:/lustre7/software/intel/oneapi/ipp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/ippcp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/ipp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/dnnl/2022.1.0/cpu_dpcpp_gpu_dpcpp/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/gdb/intel64/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/libipt/intel64/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/dep/lib:/lustre7/software/intel/oneapi/dal/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib/x64:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib/oclfpga/host/linux64/lib:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/compiler/lib/intel64_lin:/lustre7/software/intel/oneapi/ccl/2021.6.0/lib/cpu_gpu_dpcpp:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/lib/release:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/ipp/lib/intel64:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/tbb/lib/intel64/gcc4.7:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/tbb/lib/intel64/gcc4.7:/opt/pkg/intel/parallel_studio_xe_2019/update3/debugger_2019/libipt/intel64/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/daal/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/home/iceplant4561/anaconda3/lib/
2022-11-02 14:09:10.845543: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/iceplant4561/local/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib64/R/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib:/home/iceplant4561/local/lib64:/home/iceplant4561/local/lib64/R/lib:/home/iceplant4561/local/lib64:/opt/pkg/gcc/9.2.0/lib:/opt/pkg/gcc/9.2.0/lib64:/lustre7/software/intel/oneapi/vpl/2022.1.0/lib:/lustre7/software/intel/oneapi/tbb/2021.6.0/env/../lib/intel64/gcc4.8:/lustre7/software/intel/oneapi/mpi/2021.6.0//libfabric/lib:/lustre7/software/intel/oneapi/mpi/2021.6.0//lib/release:/lustre7/software/intel/oneapi/mpi/2021.6.0//lib:/lustre7/software/intel/oneapi/mkl/2022.1.0/lib/intel64:/lustre7/software/intel/oneapi/itac/2021.6.0/slib:/lustre7/software/intel/oneapi/ipp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/ippcp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/ipp/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/dnnl/2022.1.0/cpu_dpcpp_gpu_dpcpp/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/gdb/intel64/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/libipt/intel64/lib:/lustre7/software/intel/oneapi/debugger/2021.6.0/dep/lib:/lustre7/software/intel/oneapi/dal/2021.6.0/lib/intel64:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib/x64:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/lib/oclfpga/host/linux64/lib:/lustre7/software/intel/oneapi/compiler/2022.1.0/linux/compiler/lib/intel64_lin:/lustre7/software/intel/oneapi/ccl/2021.6.0/lib/cpu_gpu_dpcpp:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/libfabric/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/lib/release:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mpi/intel64/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/ipp/lib/intel64:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/tbb/lib/intel64/gcc4.7:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/tbb/lib/intel64/gcc4.7:/opt/pkg/intel/parallel_studio_xe_2019/update3/debugger_2019/libipt/intel64/lib:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/daal/lib/intel64_lin:/opt/pkg/intel/parallel_studio_xe_2019/update3/compilers_and_libraries_2019.3.199/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/home/iceplant4561/anaconda3/lib/
2022-11-02 14:09:10.845569: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

今回は、anaconda環境を駆使してtensorflowを使えるようにする
なお、国立遺伝学研究所が有するスーパーコンピューターシステムを利用していることを前提とする。

https://sc.ddbj.nig.ac.jp/

遺伝研スーパーコンピュータシステム | NIG supercomputer

自前のPCを使う場合は、GPU環境をあらかじめ構築しておく必要がある。

conda環境の構築
conda create -n GPU
conda activate GPU
conda install python=3.6.6 #python3.6がうまく動いている印象がある

その10　tensorflowのインストール

conda install cudnn=7.6.5 -y
conda install cudatoolkit=10.1 -y

#https://anaconda.org/anaconda/cudatoolkit

#使うtensorflowのバージョンによって用いるcudaが異なるようである
#cudaとは

CUDAとは - IT用語辞典

#cuda , tensorflowの互換性

【４】UbuntuにAnacondaで構築するTensorFlow-GPU環境構築 〜CUDA、cuDNNインストール編〜 - Qiita

TensorFlow2.1でCuda10.1なのにGPUが認識されない問題の解決法

condaを使用してcudaとcuDNNを複数バージョン共存して使用できるか確かめてみた | ゆるいDeep Learningcondaを使用してcudaとcuDNNを複数バージョン共存して使用できるか確かめてみた

cuda+ubuntu18.04+anaconda+tensorflow - Qiita

conda installしたcuda-toolkitとcuDNNでTensorFlowを動かす - Qiita

pip install biopython
pip install numpy==1.16.0
pip install keras==2.2.4
pip install tensorflow==2.3.0
pip install tensorflow-gpu==2.3.0 #tensorflowと合わせる
pip install pandas
pip install Bio

#↑これらの組み合わせはいくつかのサイトをみて最終的に決めた

keras.models.load_model ValueError: Unknown layer: Functional

けしからん画像分類器を作ってみる (9) 推論

#Python関数のimport時に以下のように指定する

from tensorflow.keras.preprocessing.sequence import pad_sequence

#また、モデルの学習をする場合は、
モデルのロードに使う関数を以下のように書き換える

model = tf.keras.models.load_model('Model_A.h5')

#参考までに