I am struggling on that one !
I have a script that runs prepocessing for all my subjects and runs (x3) in an fMRI study.
We run a multi-echo BOLD-ASL sequence.
Because I was inspired by AFNI_proc script, the script I use is in tcsh.
I ran very similar scripts with no problem before, but it seems that sometimes there are some parameters that will break tedana’s execution.
I don’t know precisely what is meant by module or function or script in tedana, as I’m not that familiar with python’s vocabulary, but it seems the execution is broken because the script is looking for the wrong type of object, or in the wrong folder ?
Or is it the symbolic link ?
As for an example, I copy paste the lines that are executed in the script (which are the ones I pasted and executed in terminal which actually worked) :
tedana -d BOLD_ECHO1.r01.nii BOLD_ECHO2.r01.nii BOLD_ECHO3.r01.nii BOLD_ECHO4.r01.nii -e 9.1 25 39.6 54.3 --fittype loglin --out-dir ./MEICAtoBEerased --mask mask_epi_anat.S1+tlrc.HEAD --seed 69
Tricky, especially since you say things have worked before. Can you paste in the full error message that you get? It would also be useful if you copied and pasted (or uploaded, if you can?) the script that you ran that failed.
And did you run these in the same terminal? As in - you ran the script, got the error, and then ran that bit of tedana code there and it worked? Or did you open a new terminal, etc?
Thanks for your help.
The full error message is :
``tedana -d pb03.ArDC.r01.e01.volreg+tlrc.HEAD pb03.ArDC.r01.e02.volreg+tlrc.HEAD pb03.ArDC.r01.e03.volreg+tlrc.HEAD pb03.ArDC.r01.e04.volreg+tlrc.HEAD -e 9.1 25.0 39.6 54.3 --mask mask_epi_anat.S2+tlrc.HEAD --out-dir tedana_r01
Traceback (most recent call last):
File “/home/simon/.local/bin/tedana”, line 5, in
from tedana.workflows.tedana import _main
ModuleNotFoundError: No module named ‘tedana’``
Indeed, if I run the command pasted above in the same terminal it works
The script is very long, I will skip the previous step that imply dataprocessing to leave solely steps that change directories or set variables :
#!/bin/tcsh -xef
echo "auto-generated by afni_proc.py, Tue Nov 29 09:27:30 2022"
echo "(version 7.12, April 14, 2020)"
echo "execution started: `date`"
# to execute via tcsh:
# tcsh -xef procHRF.Sujet1 |& tee output.proc.Sujet1
# to execute via bash:
# tcsh -xef commonProcLaptop.tapping 2>&1 | tee output.commonProcLaptop.tapping
set dataFolder = '/home/simon/ownCloud/MATLAB/Data/Simon/data/fMRI_data'
set AnalysisFolder = '/home/simon/Documents/CMRO2/data'
# assign output directory name
set subjects = ('ArDC' ) #'BeMa' 'CoVB' 'HiCh' 'MaKi' 'NaCa' 'RaZi')
# =========================== block: setup ============================
# script setup
# take note of the AFNI version
afni -ver
# check that the current AFNI version is recent enough
afni_history -check_date 27 Jun 2019
if ( $status ) then
echo "** this script requires newer AFNI binaries (than 27 Jun 2019)"
echo " (consider: @update.afni.binaries -defaults)"
exit
endif
cd $dataFolder
set home = $PWD
# ============= SUBJECT loop =====================
# the user may specify a single subject to run with
foreach subj ( $subjects )
set output_dir = $AnalysisFolder/$subj/common_processing
cd $subj
# verify that the results directory does not yet exist
if ( -d $output_dir ) then
echo output dir "$output_dir" already exists
exit
endif
mkdir -p $output_dir/stimuli
# set list of runs
set runs = (`count -digits 2 1 3`)
# note 4 echoes and registration echo index
set echo_list = (`count -digits 2 1 4`)
set echo_times = ( 9.1 25.0 39.6 54.3 )
set fave_echo = '01'
[...] *copying data to output folder* [...]
cd $output_dir
set TR1 = "`3dinfo -nt pb00.$subj.r01.e01.tcat+orig.HEAD`"
set TR2 = "`3dinfo -nt pb00.$subj.r02.e01.tcat+orig.HEAD`"
set TR3 = "`3dinfo -nt pb00.$subj.r03.e01.tcat+orig.HEAD`"
#set TR1end = "`expr $TR1 - 1`"
# and make note of repetitions (TRs) per run
set tr_counts=("$TR1" "$TR2" "$TR3")
[...] *outcount block* [...]
cat outcount.r*.1D > outcount_rall.1D
# get run number and TR index for minimum outlier volume
set minindex = `3dTstat -argmin -prefix - outcount_rall.1D\'`
set ovals = ( `1d_tool.py -set_run_lengths $tr_counts \
-index_to_run_tr $minindex` )
# save run and TR indices for extraction of vr_base_min_outlier
set minoutrun = $ovals[1]
set minouttr = $ovals[2]
echo "min outlier: run $minoutrun, TR $minouttr" | tee out.min_outlier.txt
[...] *despike / tshift / align / tlrc / volreg / mask* [...]
# ================================ combine =================================
# combine multi-echo data per run, using m_tedana_OC (tedana from MEICA group)
# ----- method m_tedana_OC : generate tedana (MEICA group) results -----
# (get MEICA tedana OC result, desc-optcomDenoised_bold.nii.gz)
# first run tedana commands, to see if they all succeed
foreach run ( $runs )
tedana -d pb03.$subj.r$run.e*.volreg+tlrc.HEAD \
-e $echo_times \
--mask mask_epi_anat.$subj+tlrc.HEAD \
--out-dir tedana_r$run
end
# now get the tedana results
foreach run ( $runs )
# copy result back here
3dcopy tedana_r$run/desc-optcomDenoised_bold.nii.gz pb04.$subj.r$run.combine+tlrc
end
Update :
I wrote a little script to write the tedana command in a .sh file, to run as bash (maybe the problem is purely tcsh related).
The problems remains the same : Traceback (most recent call last): File "/home/simon/.local/bin/tedana", line 5, in <module> from tedana.workflows.tedana import _main ModuleNotFoundError: No module named 'tedana'
The error appears to suggest you don’t have tedana installed, which is odd, because you say you can run it no problem in the same command. And if you didn’t have it installed, then this folder wouldn’t exist /home/simon/.local/bin/tedana
Did you pip install tedana? or is that the location that you unpacked it in? Little confused on how the tedana installation was done and despite being a tedana contributor, some of the nuances of python methods remain mysterious to me.
I’m not quite sure either - feel like i’m missing something obvious. @tsalo - you’re a wizard, ever seen something like this? How could it get into tedana.workflows.tedana and not know what tedana is?
What does which tedana show on the command line?
and can you try adding in which tedana to your script, just before where you would call it? Or even, as a very early line, just to see what it says.
Curious what that would show.
‘which tedana’ gives /home/simon/.local/bin/tedana in terminal
and /home/simon/.local/bin/tedana in tcsh script which ever the line I put it in.
That is puzzling for me.
I think the problem occurred after few weeks of leaving this script alone and trying to install some unrelated python project from github.
I was trying to get some specific version of python in an environment, and I might have done something wrong
Maybe something related ?
If you have AFNI on that system (which it sounds like you do), you could run:
afni_system_check.py -check_all
and that will provide information on all of your available Pythons (as well as other programs), which might provide some help. Though, that isn’t tedana-specific.
-------------------------------- general ---------------------------------
architecture: 64bit ELF
cpu type: x86_64
system: Linux
release: 5.19.0-46-generic
version: #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2
distribution: ubuntu 22.04 Jammy Jellyfish
number of CPUs: 16
apparent login shell: bash
shell RC file: .bashrc (exists)
--------------------- AFNI and related program tests ---------------------
which afni : /home/simon/abin/afni
afni version : Precompiled binary linux_ubuntu_16_64: Jan 6 2023
: AFNI_23.0.00 'Commodus'
AFNI_version.txt : AFNI_23.0.00, linux_ubuntu_16_64, Jan 06 2023
which python : /usr/bin/python
python version : 3.10.6
which R : /usr/bin/R
R version : R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
which tcsh : /usr/bin/tcsh
instances of various programs found in PATH:
afni : 1 (/home/simon/abin/afni)
R : 1 (/usr/bin/R)
python : 1 (/usr/bin/python3.10)
python2 : 1 (/usr/bin/python2.7)
python3 : 1 (/usr/bin/python3.10)
testing ability to start various programs...
afni : success
suma : success
3dSkullStrip : success
uber_subject.py : success
3dAllineate : success
3dRSFC : success
SurfMesh : success
3dClustSim : success
3dMVM : success
checking for R packages...
rPkgsInstall -pkgs ALL -check : success
R RHOME : /usr/lib/R
checking for $HOME files...
.afnirc : found
.sumarc : found
.afni/help/all_progs.COMP : found
------------------------------ python libs -------------------------------
** failed to load module PyQt4
-- PyQt4 is no longer needed for an AFNI bootcamp
++ module loaded: matplotlib.pyplot
module file : /home/simon/.local/lib/python3.10/site-packages/matplotlib/pyplot.py
matplotlib version : 3.6.2
-- python binaries under /usr/local/bin:
/usr/local/bin/python3.6
-------------------------------- env vars --------------------------------
PATH = /home/simon/.local/bin:/home/simon/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/simon/abin
PYTHONPATH =
R_LIBS = /home/simon/R
LD_LIBRARY_PATH =
DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =
------------------------------ data checks -------------------------------
data dir : found AFNI_data6 under $HOME
top history: ...Nov 2022 [rickr]: s05: added align_unifize_epi local
data dir : found AFNI_demos under $HOME
top history: ...ct 2020 [taylorp]: updated scripts under FATCAT_DEMO
data dir : found suma_demo under $HOME
top history: ...s_New/data/Build_tmp on Mon Mar 4 11:56:45 EST 2013
data dir : found afni_handouts under $HOME
atlas : found TT_N27+tlrc under /home/simon/abin
------------------------------ OS specific -------------------------------
which apt-get : /usr/bin/apt-get
apt-get version : apt 2.4.9 (amd64)
have Ubuntu system: ubuntu 22.04 Jammy Jellyfish
have Ubuntu afni : Precompiled binary linux_ubuntu_16_64: Jan 6 2023
========================= summary, please fix: =========================
* just be aware: login shell 'bash', but our code examples use 'tcsh'
Both your python and python3 point to the same place, which is probably a symlink. What is
ls -l `which python`
? Is that to the one in ~/.local/? You also have python libraries in /usr/local/bin/python3.6, even though your used Python looks like Python 3.10. That seems odd, and might be an issue?
A more modern version of AFNI would tell more about dot-file setup, like if you have a ~/.tcshrc file that isn’t sourcing a ~/.bashrc one, for example. To see if your shells have different Pythons, you could check the output of:
The several versions of python were nedded at some point, in order to have a specific version of a package that works with the rest of the library. It might be the issue indeed.
so, the result of ```ls -l `which python```` gives :
lrwxrwxrwx 1 root root 24 avril 11 18:06 /usr/bin/python → /etc/alternatives/python
bash -c 'which python'and tcsh -c 'which python'
point to the same folder : /usr/bin/python
The issue might be indeed that the python 3. does not contain any tedana, and for some reason, in the script it starts looking for python 3.6 ?
Should I uninstall Python 3.6 ?
Or fiddle with the symbolic links ?
I would probably try to do a bit more digging before unlinking and removing—it sounds pretty tangled at the moment, and might make things more difficult. (Python does present challenges at times with a large amount of version-dependency on subpackages—like not just needing “ver>=something”, but “ver must-be-and-only-can-be 3.14.15.9265.3.5”, which can lead to inherent mismatches across multi-dependent programs. Sigh. Note that I guess Python 3.7 has even officially reached its end-of-life date now—not sure what this all means for longterm reproducible code??? But I digress.)
At least your
ls -l `which python`
and each shell’s which python appear to all point to the same some: /usr/bin/python. That itself is a symlink, which is probably normal. Though, I have never heard of /etc/alternatives/python being used. That seems like it might have been set by some program/installation thing, and might be worth tracking down. On my Ubuntu, the above ls ... shows that /usr/bin/python points to python3, which I think would be more expected.
(Sidenote: I now generally use miniconda to manage Python libraries, because they are easy to find and each environment is modular from both the system and each other. So, the above comment I made is when I have deactivated my conda environments. It is possible setting up miniconda and having a default environment across your shells will help overcome this—but that depends on whether you have a program you need to use that requires a specific Python version number.)
Maybe you should check and see if any links/aliases are set in your terminal setup files:
I haven’t heard someone who has the same vision on Python, I feel a bit more normal. THanks for the conda advice. It is driving me insane.
Now that you mention it, I may digress a bit myself :
The core problem here, as I mentioned in earlier messages is that I tried to use a github deep learning project (from 2018) that required such specific and narrow versions of packages that I had to install python 3.6… That didn’t work out either, as some stuff are not compatible with recent ubuntu…
So indeed, I might have fiddle with some symbolic links, that might ended up with the “/alternative/python” link.
I wouldn’t know how to trace the path followed through my tcsh script, but It would be interesting to know at which step it is pointing to the wrong Python version (the one without tedana).
I was considering installing/enabling tedana in all versions of python, but I guess we’ll get back to step one, with some dependencies being broken…
The command line you proposed didn’t find any file python file with .*rc
here is the output of
ls .*rc
-rw-rw-r-- 1 simon simon 363 janv. 5 2023 .zshrc
-rw-rw-r-- 1 simon simon 149 janv. 7 17:42 .tcshrc
-rw-rw-r-- 1 simon simon 484 janv. 7 18:01 .cshrc
-rw-r--r-- 1 simon simon 4358 janv. 7 18:01 .bashrc
-rw-rw-r-- 1 simon simon 21691 janv. 8 14:39 .sumarc
-rw-rw-r-- 1 simon simon 347 avril 21 15:19 .nvidia-settings-rc ```
Adding some info.
Here is what I did do point to the python I needed when I was trying different versions :
sudo update-alternatives --config python
# My output in this case:
[sudo] password for m93:
There are 3 choices for the alternative python (providing /usr/bin/python).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/bin/python3.6 2 auto mode
* 1 /usr/bin/python2.7 1 manual mode
2 /usr/bin/python3.5 1 manual mode
3 /usr/bin/python3.6 2 manual mode
This is where the alternatives come from.
Do you know a secure and sure way to change this link ?
I think i’m not gonna interfere with python anymore on this computer and will use another one. So it doesn’t matter if it is a little more difficult to change the link target
Thanks for your help in any case
P.S : after following the links,
usr/bin/python points to etc/alternatives/python
which itself points to usr/bin/python3,
which points to usr/bin/python3.10
OK. The grep .. command that I had suggested was to see if ‘python’ was being referred to in any of your terminal setup files—the fact that no text was output means that none is. So there aren’t any aliases there, for example. That is fine.
I have to admit that I have never heard the ‘alternatives’ stuff. Eek. I can’t really offer any helpful suggestions about that, I’m afraid. It is odd to see (the same?) python3.6 appear twice there, just with different modes. Ummm, I just tried searching a bit online and this suggests you could alter your choice by doing the same kind of command there, getting prompts:
… but again, I don’t know. Is there a local support person you could ask?
I wonder if the easiest might be to try going with miniconda, and making a default environment with it. I think that should become first in line, and then the Python confusion might be avoidable. And if that doesn’t work, it is essentially all sandboxed in a directory you can just get rid of, removing that directory and reference to it in your ~/.*rc files. Here are some notes on miniconda on the AFNI docs: https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/miniconda.html
… though at this exact moment our server is down for some reason.
Thanks Peter.
From what you said, i’ll try different ways, from the softest one (conda) to the most barbarian (uninstall python to reinstall it) until it works again.
I’ve got a second computer at my lab where things work fine (ubuntu 18.X) so in any case I can still run all scripts heavy on this machin.
