General test harness improvements

[natanlao]

A bunch of miscellaneous improvements to the test harness.

• Upgrade to treon v0.1.3. This lets us remove some of the workaround code we have in repo. (Namely, running notebooks in the directories they live in, and ignoring certain notebooks.)
• Use Python 3.8 instead of 3.8-dev for Travis builds.
• Cache pip downloads between Travis builds.
• Fix numpy install failing on 3.8
• Fix output testing failing silently

This PR closes #89.

As a consequence of some of the changes above, build time has been cut from some 30 minutes to just over three minutes. Neat!

[chmreid]

a few suggestions, but overall looks okay to me!

[chmreid]

Travis is failing for Python 3.8 only, due to some packages that are not imported.

It looks like there is a problem with Python 3.8 and the llvmlite package. It fails to build, which causes other packages to not be installed and therefore causes import errors. (I'm also experiencing some similar errors in my build for #87.) Relevant sections from your Travis output:

  Building wheel for llvmlite (setup.py) ... error

  ERROR: Command errored out with exit status 1:

   command: /home/travis/virtualenv/python3.8.0/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6s5nbpy4/llvmlite/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6s5nbpy4/llvmlite/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-cz7zdme7 --python-tag cp38

       cwd: /tmp/pip-install-6s5nbpy4/llvmlite/

  Complete output (23 lines):

  running bdist_wheel

  /home/travis/virtualenv/python3.8.0/bin/python /tmp/pip-install-6s5nbpy4/llvmlite/ffi/build.py

  LLVM version... # static-libstdc++ avoids runtime dependencies on a

  # particular libstdc++ version.

  g++  -shared   -fPIC -I/usr/local/clang-7.0.0/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -ffunction-sections -fdata-sections -O3 -DNDEBUG -fno-exceptions -fno-rtti -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -fno-rtti -g -flto assembly.cpp bitcode.cpp core.cpp initfini.cpp module.cpp value.cpp executionengine.cpp transforms.cpp passmanagers.cpp targets.cpp dylib.cpp linker.cpp object_file.cpp -o libllvmlite.so  -L/usr/local/clang-7.0.0/lib -flto -Wl,--exclude-libs=ALL -lLLVMLTO -lLLVMPasses -lLLVMObjCARCOpts -lLLVMSymbolize -lLLVMDebugInfoPDB -lLLVMDebugInfoDWARF -lLLVMMIRParser -lLLVMFuzzMutate -lLLVMCoverage -lLLVMTableGen -lLLVMDlltoolDriver -lLLVMOrcJIT -lLLVMXCoreDisassembler -lLLVMXCoreCodeGen -lLLVMXCoreDesc -lLLVMXCoreInfo -lLLVMXCoreAsmPrinter -lLLVMSystemZDisassembler -lLLVMSystemZCodeGen -lLLVMSystemZAsmParser -lLLVMSystemZDesc -lLLVMSystemZInfo -lLLVMSystemZAsmPrinter -lLLVMSparcDisassembler -lLLVMSparcCodeGen -lLLVMSparcAsmParser -lLLVMSparcDesc -lLLVMSparcInfo -lLLVMSparcAsmPrinter -lLLVMPowerPCDisassembler -lLLVMPowerPCCodeGen -lLLVMPowerPCAsmParser -lLLVMPowerPCDesc -lLLVMPowerPCInfo -lLLVMPowerPCAsmPrinter -lLLVMNVPTXCodeGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMNVPTXAsmPrinter -lLLVMMSP430CodeGen -lLLVMMSP430Desc -lLLVMMSP430Info -lLLVMMSP430AsmPrinter -lLLVMMipsDisassembler -lLLVMMipsCodeGen -lLLVMMipsAsmParser -lLLVMMipsDesc -lLLVMMipsInfo -lLLVMMipsAsmPrinter -lLLVMLanaiDisassembler -lLLVMLanaiCodeGen -lLLVMLanaiAsmParser -lLLVMLanaiDesc -lLLVMLanaiAsmPrinter -lLLVMLanaiInfo -lLLVMHexagonDisassembler -lLLVMHexagonCodeGen -lLLVMHexagonAsmParser -lLLVMHexagonDesc -lLLVMHexagonInfo -lLLVMBPFDisassembler -lLLVMBPFCodeGen -lLLVMBPFAsmParser -lLLVMBPFDesc -lLLVMBPFInfo -lLLVMBPFAsmPrinter -lLLVMARMDisassembler -lLLVMARMCodeGen -lLLVMARMAsmParser -lLLVMARMDesc -lLLVMARMInfo -lLLVMARMAsmPrinter -lLLVMARMUtils -lLLVMAMDGPUDisassembler -lLLVMAMDGPUCodeGen -lLLVMAMDGPUAsmParser -lLLVMAMDGPUDesc -lLLVMAMDGPUInfo -lLLVMAMDGPUAsmPrinter -lLLVMAMDGPUUtils -lLLVMAArch64Disassembler -lLLVMAArch64CodeGen -lLLVMAArch64AsmParser -lLLVMAArch64Desc -lLLVMAArch64Info -lLLVMAArch64AsmPrinter -lLLVMAArch64Utils -lLLVMObjectYAML -lLLVMLibDriver -lLLVMOption -lLLVMWindowsManifest -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMLineEditor -lLLVMInterpreter -lLLVMExecutionEngine -lLLVMRuntimeDyld -lLLVMCodeGen -lLLVMTarget -lLLVMCoroutines -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMScalarOpts -lLLVMLinker -lLLVMIRReader -lLLVMAsmParser -lLLVMInstCombine -lLLVMBitWriter -lLLVMAggressiveInstCombine -lLLVMTransformUtils -lLLVMAnalysis -lLLVMProfileData -lLLVMObject -lLLVMMCParser -lLLVMMC -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMBitReader -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -lrt -ldl -lpthread -lm

  g++: error: unrecognized command line option ‘-Wcovered-switch-default’

  g++: error: unrecognized command line option ‘-Wstring-conversion’

  Makefile.linux:20: recipe for target 'libllvmlite.so' failed

  make: *** [libllvmlite.so] Error 1

  7.0.0

  

  SVML not detected

  Traceback (most recent call last):

    File "/tmp/pip-install-6s5nbpy4/llvmlite/ffi/build.py", line 168, in <module>

      main()

    File "/tmp/pip-install-6s5nbpy4/llvmlite/ffi/build.py", line 158, in main

      main_posix('linux', '.so')

    File "/tmp/pip-install-6s5nbpy4/llvmlite/ffi/build.py", line 150, in main_posix

      subprocess.check_call(['make', '-f', makefile])

    File "/opt/python/3.8.0/lib/python3.8/subprocess.py", line 364, in check_call

      raise CalledProcessError(retcode, cmd)

  subprocess.CalledProcessError: Command '['make', '-f', 'Makefile.linux']' returned non-zero exit status 2.

  error: command '/home/travis/virtualenv/python3.8.0/bin/python' failed with exit status 1

  ----------------------------------------

  ERROR: Failed building wheel for llvmlite

  Running setup.py clean for llvmlite

  Building wheel for numexpr (setup.py) ... done

  Created wheel for numexpr: filename=numexpr-2.7.0-cp38-cp38-linux_x86_64.whl size=407297 sha256=d099b21950a3a1f9194196cd751ca4ce3ae5480f7838c4083289f59d9d138175

  Stored in directory: /home/travis/.cache/pip/wheels/97/66/f3/1f0af5345b0d44368b7d82539b6e15003c500107c761e0312a

Successfully built loompy louvain python-igraph numba numpy-groupies umap-learn numexpr

Failed to build llvmlite

[natanlao]

It looks like llvmlite does not support Python 3.8 (numba/llvmlite#527). Why this didn't break testing, I can't say.

The issue indicates that Python 3.8 support should be readily available over pip some time after New Year's. I think for now it makes sense to drop 3.8 testing and add it after curtailment.

[chmreid]

agreed, let's drop 3.8 for now and add it back in once everything in the build chain works with 3.8

[chmreid]

🎉
🎈🎈🎈

[chmreid]

ERROR: scanpy 1.4.4.post1 has requirement matplotlib==3.0.*, but you'll have matplotlib 3.1.2 which is incompatible.

link

☝️ still have a bit of red text during the pip install phase - but let's clear that up in a later pull request. LGTM 💪