This problem could be solved by reinstalling CUDA, CUDNN, and NVIDIA Drivers. It was not known what suddenly happened to the normal Conda environment, but it is expected that a conflict between internal drivers may have been the cause.
First, when I deleted the driver with apt-get purge, there was a remaining file, so I had to search and remove it one by one with dpkg -l | grep. And to match the compatibility between CUDA versions, nvidia-driver 510, cuda 11.6, and cudnn 8.4.0.27 versions were installed. When installing dev from the NVIDIA official website, the latest version is installed, so I installed it through runfile. At this time, nvidia-driver should not be installed together, and nvidia-driver for CUDA 11.6 version had to be reinstalled separately. When installing 470 and 520 versions with different nvidia-driver versions, the existing problem could not be solved.