Conda environment not playing nice with iograft

The RNA Sequencing conda environment I build is installing Python 2.7 but I think it interferes with iograft internal Python3 calls.

Here is the conda environment YAML file’s content:

I am happy to trim back dependencies but fastqc and multiqc are the bare minimum I must have.

name: rnaseq
channels:
  - bioconda
dependencies:
  - cromwell
  - cromshell
  - pip
  - tophat
  - fastqc
  - multiqc
  - star
  - trim-galore
  - samtools
  - trimmomatic
  - picard
  - nodejs
  - wdltool
  - subread
  - minimap2
  - pip:
    - cwlref-runner

Hey Nicholas,

iograft can work with Python 2.7, but it requires a new iograft environment with a custom “subcore command” that calls Conda’s Python interpreter rather than iograft’s built-in interpreter. This is how many of our other integrations work as well (i.e. Maya 2020 with Python 2.7; Blender with Python 3.10, etc).

The first step would be creating a new iograft environment to represent the Conda environment. The key items in this iograft environment would be to:

  • Update the Path setting to include the directory containing the Python executable your Conda environment is using.
  • Update the Python Path setting to include any directories that should be available in this Conda environment. At a minimum this should include the “python27” subdirectory and the “types” subdirectory from the iograft installation.
  • Update the Subcore Launch Command.

The last step is most likely the item that is causing the confusion of how iograft works with other Python interpreters. By default, iograft will use the built-in “iograft_subcore” executable which in fact uses iograft’s built-in Python 3 interpreter rather than the desired Python interpreter of your Conda environment in this case.

To ensure iograft uses the expected Python interpreter, you will need to use a different Subcore launch command. There is an example subcore command in the iograft GitHub repository that might get you the behavior you are looking for with Conda (if Conda’s python executable directory is added to the Path as mentioned above): iograft/bin at main · iograft/iograft · GitHub

That being said, to fully “activate” a Conda environment that is running within iograft, you will most likely need a separate subcore command that executes the full conda activate <ENV_NAME> that you would normally run when working with Conda.

I will work to put together an example with Conda in iograft, but in the meantime if you are able to build the iograft environment and share the JSON for that environment, we can make sure it all looks good. (To access the environment’s JSON within iograft, go to “Help > Open User Settings Folder” and then navigate to the “envs” subdirectory. There should be a JSON file in that directory corresponding to the environment you have built).

1 Like

I managed to get a Conda environment working within iograft using the “iograftpy_subcore” that I mentioned previously (and was able to do a quick proof of concept importing multiqc).

Here is my iograft environment JSON:

{
    "plugin_path": [
        "/Applications/iograft.app/Contents/types",
        "/Applications/iograft.app/Contents/nodes",
        "/Users/knowlonix/.iograft/types",
        "/Users/knowlonix/.iograft/nodes",
        "/Users/knowlonix/opt/anaconda3/envs/test_conda/iograft_nodes"
    ],
    "subcore": {
        "launch_command": "iograftpy_subcore"
    },
    "path": [
        "/Applications/iograft.app/Contents/bin",
        "/Users/knowlonix/Projects/iograft-public/iograft/bin",
        "/Users/knowlonix/opt/anaconda3/envs/test_conda/bin"
    ],
    "python_path": [
        "/Applications/iograft.app/Contents/types",
        "/Applications/iograft.app/Contents/python27"
    ],
    "environment_variables": {},
    "appended_environments": [],
    "name": "conda_test"
}

Besides what I mentioned previously, in order to use the iograftpy_subcore, I have also cloned the “iograft” GitHub repository and added the “bin” directory from that repository to the environment’s Path.

And here was my quick test (all the test_multiqc node is doing is importing the multiqc package from the Conda environment and outputting the package location):

Let me know if this gets you going or if you have any other questions for now!

1 Like

Here is my attempt at updating the environment and also cloning the github iograft repo to access the bin directory

NOTE : I am running Ubuntu 20.04

// Environment settings for iograft. These settings define an environment
// within the iograft system.

// WARNING: This file is auto-generated so comments and unrecognized
// settings will NOT be preserved.
{
    "plugin_path": [
        "/opt/iograft/types",
        "/opt/iograft/nodes",
        "/home/nyue/.iograft/types",
        "/home/nyue/.iograft/nodes",
        "/home/nyue/projects/iograft/iograft_git/python/rnaseq"
    ],
    "subcore": {
        "launch_command": "iograftpy_subcore"
    },
    "path": [
        "/opt/iograft/bin",
        "/home/nyue/anaconda3/envs/rnaseq/bin",
        "/home/nyue/projects/iograft/github_iograft_git/bin"
    ],
    "python_path": [
        "/opt/iograft/types",
        "/opt/iograft/python27"
    ],
    "environment_variables": {},
    "appended_environments": [],
    "name": "rnaseq"
}

The error I get when I try to initialise the environment looks like these

[2022-08-28 18:28:26.816] [info] Launching in environment: 'default'.
[2022-08-28 18:28:28.211] [info] Starting new local session.
[2022-08-28 18:28:28.231] [info] Initializing iograft. (Version: 1.3.1)
[2022-08-28 18:28:28.231] [info] Loading embedded python from plugin: /opt/iograft/lib/iograft_embedded_py3
[2022-08-28 18:28:28.296] [info] Setting current environment to: 'default' based on the 'IOGRAFT_ENV' environment variable.
[2022-08-28 18:28:28.570] [info] Loaded plugin successfully: '/opt/iograft/types/iopythontypes.py'.
[2022-08-28 18:28:28.573] [info] Loaded plugin successfully: '/opt/iograft/types/iobasictypes.py'.
[2022-08-28 18:28:28.574] [info] Loaded plugin successfully: '/opt/iograft/types/libbasic_types.iot'.
[2022-08-28 18:28:28.595] [info] Loaded plugin successfully: '/opt/iograft/nodes/write_json_file.py'.
[2022-08-28 18:28:28.596] [info] Loaded plugin successfully: '/opt/iograft/nodes/set_file_extension.py'.
[2022-08-28 18:28:28.597] [info] Loaded plugin successfully: '/opt/iograft/nodes/list_directory.py'.
[2022-08-28 18:28:28.598] [info] Loaded plugin successfully: '/opt/iograft/nodes/create_directory.py'.
[2022-08-28 18:28:28.638] [info] Loaded plugin successfully: '/opt/iograft/nodes/copy_files.py'.
[2022-08-28 18:28:28.640] [info] Loaded plugin successfully: '/opt/iograft/nodes/write_file_lines.py'.
[2022-08-28 18:28:28.640] [info] Loaded plugin successfully: '/opt/iograft/nodes/load_json_file.py'.
[2022-08-28 18:28:28.641] [info] Loaded plugin successfully: '/opt/iograft/nodes/read_file_lines.py'.
[2022-08-28 18:28:28.642] [info] Loaded plugin successfully: '/opt/iograft/nodes/get_environment_variable.py'.
[2022-08-28 18:28:28.643] [info] Loaded plugin successfully: '/opt/iograft/nodes/remove_directory.py'.
[2022-08-28 18:28:28.644] [info] Loaded plugin successfully: '/opt/iograft/nodes/write_json_string.py'.
[2022-08-28 18:28:28.645] [info] Loaded plugin successfully: '/opt/iograft/nodes/split_string.py'.
[2022-08-28 18:28:28.646] [info] Loaded plugin successfully: '/opt/iograft/nodes/read_file.py'.
[2022-08-28 18:28:28.647] [info] Loaded plugin successfully: '/opt/iograft/nodes/split_file_extension.py'.
[2022-08-28 18:28:28.648] [info] Loaded plugin successfully: '/opt/iograft/nodes/set_dictionary_value.py'.
[2022-08-28 18:28:28.649] [info] Loaded plugin successfully: '/opt/iograft/nodes/create_empty_dictionary.py'.
[2022-08-28 18:28:28.650] [info] Loaded plugin successfully: '/opt/iograft/nodes/concatenate_strings.py'.
[2022-08-28 18:28:28.651] [info] Loaded plugin successfully: '/opt/iograft/nodes/filter_files.py'.
[2022-08-28 18:28:28.652] [info] Loaded plugin successfully: '/opt/iograft/nodes/write_file.py'.
[2022-08-28 18:28:28.653] [info] Loaded plugin successfully: '/opt/iograft/nodes/set_current_directory.py'.
[2022-08-28 18:28:28.654] [info] Loaded plugin successfully: '/opt/iograft/nodes/get_current_directory.py'.
[2022-08-28 18:28:28.655] [info] Loaded plugin successfully: '/opt/iograft/nodes/get_dictionary_value.py'.
[2022-08-28 18:28:28.656] [info] Loaded plugin successfully: '/opt/iograft/nodes/format_string.py'.
[2022-08-28 18:28:28.657] [info] Loaded plugin successfully: '/opt/iograft/nodes/load_json_string.py'.
[2022-08-28 18:28:28.658] [info] Loaded plugin successfully: '/opt/iograft/nodes/get_python_path.py'.
[2022-08-28 18:28:28.658] [info] Loaded plugin successfully: '/opt/iograft/nodes/split_path.py'.
[2022-08-28 18:28:28.659] [info] Loaded plugin successfully: '/opt/iograft/nodes/join_path.py'.
[2022-08-28 18:28:28.660] [info] Loaded plugin successfully: '/opt/iograft/nodes/string_length.py'.
[2022-08-28 18:28:28.700] [info] Loaded plugin successfully: '/opt/iograft/nodes/create_temp_dir.py'.
[2022-08-28 18:28:33.111] [info] Launching subcore process for environment: 'rnaseq'.
Traceback (most recent call last):
  File "/home/nyue/projects/iograft/github_iograft_git/bin/iograftpy_subcore", line 6, in <module>
    import iograft
  File "/opt/iograft/python27/iograft.py", line 34, in <module>
    import _iograftpy
ImportError: /opt/iograft/python27/_iograftpy.so: undefined symbol: PyUnicodeUCS2_FromString
[2022-08-28 18:28:34.115] [warning] Subcore process for env: 'rnaseq' exited with non-zero exit code.
[2022-08-28 18:28:35.115] [info] Subcore process for environment: 'rnaseq' has terminated.

Ah, thanks for the report. Currently we are building iograft’s Python 2.7 bindings on Debian with ucs2 which is likely conflicting with the ucs setting of the Python 2.7 interpreter that Conda provides.

To confirm, can you run your Conda environment’s Python interpreter directly and give the output of this script:

import sys
print(sys.maxunicode)

iograft’s Ubuntu Python 2.7 bindings are currently an outlier (all 3.x Python bindings and 2.7 Python bindings on CentOS are built with ucs4), so I will do some digging and see if we should default the Ubuntu Python 2.7 bindings to ucs4 as well.

Either way, we will either make both a ucs2 AND a ucs4 bindings available, or switch over to all ucs4 bindings on Linux. This would go out in the next release of iograft.

(rnaseq) nyue@Compaq-8200-Elite:~$ which python
/home/nyue/anaconda3/envs/rnaseq/bin/python
(rnaseq) nyue@Compaq-8200-Elite:~$ python
Python 2.7.18 |Anaconda, Inc.| (default, Jun  4 2021, 14:47:46) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.maxunicode)
1114111

Great, thanks! This confirms the conflict I had guessed.

We will be switching all platforms/Python versions fully to UCS4 in the next release, which should resolve the issue.

1 Like

The update to fully UCS4 on Ubuntu was released in version 1.3.2 today.