Skip to content

[bug] gubbins fails when run under apptainer #667

Description

@pvanheus

Description
The gubbins step of the snippy workflow fails:

[35/5c25d1] NOTE: Process `GUBBINS:GUBBINS_MODULE (core-snp)` terminated with an error exit status (1) -- Execution is retried (1)
[c4/33af0d] Re-submitted process > GUBBINS:GUBBINS_MODULE (core-snp)
ERROR ~ Error executing process > 'GUBBINS:GUBBINS_MODULE (core-snp)'

Caused by:
  Process `GUBBINS:GUBBINS_MODULE (core-snp)` terminated with an error exit status (1)


Command executed:

  if [ "true" == "true" ]; then
      gzip -c -d core-snp-clean.full.aln.gz > core-snp-clean.full.aln
  fi

  run_gubbins.py \
      --threads 6 \
      --prefix core-snp \
      --iterations 5 --min-snps 3 --min-window-size 100 --max-window-size 10000 --filter-percentage 25.0 \
      core-snp-clean.full.aln

  # Create masked alignment
  mask_gubbins_aln.py \
      --aln core-snp-clean.full.aln \
      --gff core-snp.recombination_predictions.gff \
      --out core-snp.masked.aln

  # Cleanup
  if [ "true" == "true" ]; then
      rm -rf core-snp-clean.full.aln
  fi
  gzip *.masked.aln *.embl *.fasta *.gff *.vcf

  # Move supplemental outputs to gubbins folder
  mkdir gubbins
  mv core-snp.branch_base_reconstruction.embl.gz        core-snp.recombination_predictions.embl.gz        core-snp.recombination_predictions.gff.gz        core-snp.per_branch_statistics.csv        core-snp.filtered_polymorphic_sites.fasta.gz        core-snp.filtered_polymorphic_sites.phylip        core-snp.final_tree.tre        core-snp.node_labelled.final_tree.tre        core-snp.summary_of_snp_distribution.vcf.gz        gubbins/

  cat <<-END_VERSIONS > versions.yml
  "GUBBINS:GUBBINS_MODULE":
      gubbins: $(run_gubbins.py --version 2>&1)
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  /usr/local/lib/python3.10/site-packages/gubbins/__init__.py:12: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
    import pkg_resources
  Traceback (most recent call last):
    File "/usr/local/bin/run_gubbins.py", line 7, in <module>
      from gubbins.run_gubbins import main
    File "/usr/local/lib/python3.10/site-packages/gubbins/__init__.py", line 17, in <module>
      from gubbins import common
    File "/usr/local/lib/python3.10/site-packages/gubbins/common.py", line 44, in <module>
      from gubbins.pyjar import jar, get_base_patterns
    File "/usr/local/lib/python3.10/site-packages/gubbins/pyjar.py", line 237, in <module>
      def seq_to_int(seq,out_seq):
    File "/usr/local/lib/python3.10/site-packages/numba/core/decorators.py", line 227, in wrapper
      disp.enable_caching()
    File "/usr/local/lib/python3.10/site-packages/numba/core/dispatcher.py", line 811, in enable_caching
      self._cache = FunctionCache(self.py_func)
    File "/usr/local/lib/python3.10/site-packages/numba/core/caching.py", line 687, in __init__
      self._impl = self._impl_class(py_func)
    File "/usr/local/lib/python3.10/site-packages/numba/core/caching.py", line 423, in __init__
      raise RuntimeError("cannot cache function %r: no locator available "
  RuntimeError: cannot cache function 'seq_to_int': no locator available for file '/usr/local/lib/python3.10/site-packages/gubbins/pyjar.py'

Work dir:
  /srv/pvh/cholera_analysis/work/c4/33af0d92e8ace59be1817839a9e700

Container:
  /srv/pvh/singularity_cache/depot.galaxyproject.org-singularity-gubbins-3.4.3--py310hfc0ef84_1.img

Steps to Reproduce
Steps to reproduce the behavior:

bactopia --wf snippy --reference reference/reference.gbk --bactopia cleaned_reads  --outdir bactopia_snippy

Expected Behavior
Completion of snippy workflow.

Execution Environment

  • Bactopia Version: 4.0.1
  • OS: Ubuntu 24.04
  • Environment: pixi 0.67.2, apptainer 1.5.0

Additional Information
I suspect that this is related to this: nickjcroucher/gubbins#402 (comment) - but setting the sessiondir max size = 1000 parameter like suggested does not resolve the issue. The underlying issue here is, in fact, numba, which is trying to write to the singularity container filesystem, as explained in this issue: numba/numba#4908 (comment) - setting --writeable-tmpfs seems to resolve the issue - i.e. I have increased sessiondir max size and set this flag on the singularity command line. Ideally I'd like to find a way of solving this without changing the Apptainer / Singularity configuration file, thus relocating the Numba cache with e.g. SINGULARITYENV_NUMBA_CACHE_DIR="$TMPDIR" might be a better option.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingfixed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions