2016-05-09 Exceptional meeting regarding build customization¶
Hangout link: ???
Attendees: Michael Sarahan, Ray Donnelly, Kale Franz, Michael Grant, Ullrich Koethe, Stuart Berg, Phil Elson, Mark Wiebe
* MichaelS: proposing to have dependency resolution become part of meta.yaml rendering. May require extra (3rd) jinja rendering pass.
Need to know dependencies before Jinja templates can be fully expressed (main issue in #747)
Phil: not good, because you cant differentiate between a manual entry of a value, and a templated entry of a value.
* Dont know which ones are important for build string * Build strings only matter when we have a build matrix; there to disambiguate * MichaelS proposes manual specification of entries in build string * Ullrich: if build strings just disambiguate, why not use hash? * MichaelS: hash fine, but needs to be reproducible. * Heart of issue is that matrix build tools need some way to understand what parameterizes the build matrix. * MichaelS offered that no matter what solution we settle on, there will be something readily inspectable. Maybe this can feed into automatically creating the build string.
If recipe-local config.yaml is needed, Kale asks why not include it directly in meta.yaml
* Not sure what use case necessitates recipe-local config.yaml (global one is agreed upon by all). MichaelS OK with meta.yaml modifications if need be.
Phil points out that global config.yaml might still need code changes for new versions of, say, Python.
* MichaelS says no, theres surely ways we can express things like latest 2 versions - just question of how to technically achieve.
Ray: filter things out of global config.yaml when they are not used in a recipe (so that matrix dimensions are not needlessly added)
* Ullrich: However, one must be careful to not filter out indirect dependencies that are not explicitly named in meta.yaml, but were intentionally added to config.yaml. (example: meta.yaml lists h5py as a requirement, and config.yaml restricts the indirect hdf5 dependency to a particular variant of the hdf5 package). It is safer to err on the too much side.
Build environment specification:
* Conda-build 848 has bootstrap environment: user creates an env, that env is captured as build requirements for another recipe. * Phil: this ruins reproducibility, since it depends on the setup of the users system at build time. * Stuart: were already not reproducible, and this bootstrap doesnt make things worse. * MichaelS: if this is used to hard-pin build-time dependencies, it may actually help reproducibility. Hard-pinnning at build time is orthogonal to this PR, but may be easier to achieve with it. * Ullrich: if the contents of the build environment are recorded in index.json, one can recreate the bootstrap environment at any time (or equivalently, create config.yaml from that information), subject to the degree of accuracy of the recorded data (e.g. are the source channels recorded, or only version numbers and build strings?) * MichaelS, Kale concerned that people may include too many unnecessary packages in build requirements this way, out of carelessness or ignorance * Ullrich: not a big deal - runtime dependencies are what really matter.
* Mark: 848 is OK, but how does it address compiler and build flags?
Conda 1959 is example of build flags. Uses bat/sh script at beginning of bld.bat/build.sh. Similar to Marks effort. Dont like having to add that extra shell script, though - want native support
* MichaelS offered integration with conda-build, when use cases are better understood. Some kind of hook to call standardized setup script.
Ullrich: The important aspect is that it should be easy for the conda-build user to specify which setup script to use. Having specialized build packages installed in a bootstrap environment seems to be an easy solution (e.g. conda create -n my_bootstrap_env visual-studio-build=12.0 or conda create -n my_bootstrap_env visual-studio-build=12.0.debug)
Ullrich points out that these packages also need to exclude use with other packages, as they are not compatible
Mark: would like to create easy to use instance of http://vfxplatform.com/ using conda. Wants to be in the loop as we get that functionality worked out.
Ullrich: Critical requirements for the final solution (whatever it will be):
* Build dependency customization must be possible for _any_ package, although it would be OK if it were a bit more complicated for the less common situations. It is crucial to support two cases: * Customize _indirect _dependencies not specified in meta.yaml. * Narrow dependencies listed in meta.yaml (e.g. python 3.* => python 3.5.*). * Run requirements must be specialized during the build such that version resolution will only select compatible variants, even if active channels offer newer, yet incompatible variants (right now I usually use -c ukoethe --override-channels, but this is a pain).