2016-08-12: General discussion¶
Time: 14:00 UTC
Hangout link: https://hangouts.google.com/call/v5olhwzpfzgzpoq5i3wthjpqpie
2016-07-22: General discussion
Attendees
Eric Dill
Standing items
How many repos?
How many contributors?
New core devs?
Agenda
Prerelease versions
* Python prerelease done at [conda forge/python feedstock#45](https://github.com/conda-forge/python-feedstock/pull/45) - is this an example to follow?
Do we have documentation on how to do this?
Waiting PR: conda forge/scikit image feedstock#2
Conda itself: conda/conda#3262#issuecomment-239410077
proposal for naming pre-release channels:
* embed the package name in the anaconda label so that you can specify exactly which pre-release things to install
The conda install command to specify from a label other than
main
is:** *** **`conda** install -c conda-forge/label/rc <package>` * So if you embed, for example, "matplotlib-" in the label name, then you can specifically install *just* the matplotlib pre-release with: * `conda install -c conda-forge/label/matplotlib-rc matplotlib`
Status page
* Have dependencies.
Some code for the webservice
Feedstocks philosophy: Explicit vs implicit / reproducible vs redundant
OSX - getting back to a usable, coherent, stack
* libc++ (clang) vs libstdc++ (gcc/g++)
Minimum OSX required for clang (10.8, I think?)
Actually clang is usable beginning in 10.7. So, this would be viable given your compatibility constraints.
Also, all the refs I have seen suggest that this will still have C++11 support.
Compatibility with defaults (built on 10.7, uses gcc) - where will people break? I think only if mixing packages - how do we assure that we have all the ones we need?
Improving infrastructure
* Finish out GitHub API issues ( [conda forge/conda forge.github.io#172](https://github.com/conda-forge/conda-forge.github.io/issues/172) )
Better workflows with staged-recipes
* Fast finish AppVeyor on merge ( [conda forge/staged recipes#1142](https://github.com/conda-forge/staged-recipes/pull/1142) )
Drop Travis CI matrix ( conda forge/staged recipes#1234 )
Use CircleCI for feedstock generation ( conda forge/staged recipes#916 )
Keeping recipes out of PRs ( conda forge/staged recipes#942 )
Bank work in partial conversion ( conda forge/staged recipes#915 )
Low level packaging
Basic community practices when PR-ing to staged-recipes.
No need to re-discuss this. I am still writing the docs and, if ready, I will send the link tomorrow (or after SciPy ;-)
NetCDF (
also curl/ca-certificates and Perl packages) - Done?* curl and ca-certificates are done and available.
Perl is no longer relevant as part of this process
Notifications (how do we stay on top of them)
Standardizing installs
* Mention [`toolchain`](https://github.com/conda-forge/toolchain-feedstock) . * Discuss rollout to feedstocks. * Get feedback on [`python-toolchain`](https://github.com/conda-forge/staged-recipes/pull/642)
MSYS2
* Available on defaults - was in conda 4.1.7, but that was pulled. Coming in 4.1.8.
Discussing Ray Donnelly’s work on MSYS2 packages and how we want to use and integrate these into conda-forge.
Some use cases to consider OpenBLAS, FFTW, build tools, others?
Binary data
* Do we include it in recipes?
What kinds do we allow if any (e.g. icons)?
How do we verify the licensing?
How do we verify that they are safe?
Dev releases: Where do they happen?
* Do we do them at conda-forge? * Maybe add a label. * Do we let others do them with a feedstock on their own repo?
How do we enforce whatever we decide?
Conda-forge installer
* We have Python 3.5 and 3.4 now. Would be nice to complete Python 2.7.
Have all dependencies. Though
conda-build
has some kinks to be worked out.Many open questions about the installer including its name
Where do we host the installers? Git tags?
This can work right now if you pin to conda-build 1.21.7
But, is realistically blocked due to a setuptools entrypoints issue on windows that is fixed with conda 4.2, but 4.2 is not released yet. conda 4.2 is slated to be released by the end of August
Channel mirroring
* Can this point be a little bit explained? I thought about this as well and would like to contribute to this point. * Eric Dill has put together a script for copying a package from one channel to another here: [conda forge/conda forge.github.io#134](https://github.com/conda-forge/conda-forge.github.io/pull/134) * I have a really, really crude script that copies all of the packages in one channel to another that I just put at: [](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555)[https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555) * conda-build-all can copy from one channel to another: `conda build-all --inspect-channels conda-forge --upload-channels astropy some_packge_recipe` will copy the `some_package` from the channel conda-forge to astropy if it can, or build it if it doesn't exist on conda-forge. Discussion about what the desired behavior should be has started at: [SciTools/conda build all#46](https://github.com/SciTools/conda-build-all/issues/46)
Feedstock history
* Is it sacred?
Do we rebase/force push?
* If so, under what conditions?
How do we avoid multiple people doing this simultaneously?
* I don't think you can. * IMHO, if it's just one author in staged recipes, sure. If feedstock, no force push - only to PRs to feedstock. If people don't mind merge PRs, it sure is a lot simpler to not rebase. I have messed up rebasing a few times recently... =(
Docker hosting solution
* Docker Hub builds were broken for a week and a half.
Have switched to quay.io currently.
Mirroring quay.io image on Docker Hub.
Thoughts about quay.io? Thoughts about hosting in general?
Continuum metadata request: can we add these to linter?
* example metadata: [](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)[https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)
Also, distinguish summary (limit of 77 or 80 chars) from description (unlimited)
Anaconda verify: would be nice to meet in the middle, rather than diverge. conda-build may integrate anaconda-verify, would be nice if conda-forge added metadata here.
Google hangouts has a max capacity of 10. Is it worth considering other methods of communication so everyone who wants to participate can?
Maybe this ( http://www.freeconferencecalling.com/ ) is an option.
Bluejeans
Continuum has webex. Past experience is that some Linux platforms had trouble connecting
Drop numpy 1.10 and reduce our build matrix. (Numba now works with numpy 1.11.)
This comment from the PR for graphviz is the best summary I’ve seen: conda forge/staged recipes#568#issuecomment-225315370
Thanks for pointing this out. The described solution looks reasonable and is preferable to prefixing package names. Great!
What is the benefit?
Will we distinguish between libs and standalone tools, similar to Debian? I would strongly suggest to do this, because it is (1) established and (2) more accessible for the user (if he wants to use a library, he knows the language. If he wants to use a standalone, he doesn’t care). ( )https://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html#s-package_names)
Will there be an orchestrated move? If not, how do we deal with inconsistencies and potential conflicts (installing both python-h5py and h5py).
* we will probably go with meta-packages for conflicting packages
Signing packages
Should be easy to do. ( http://conda.pydata.org/docs/signed-packages.html )
There has been some interest previously.
HTTPError: 503 Server Error: Service Unavailable: Back-end server is at capacity for url…
Seems we are regularly running into this issue under normal usage conditions.
Had discussed previously caching packages on AppVeyor and trying to reuse those to start.
Maybe we need to consider caching on all CIs.
Building our own Miniconda-like self-extracting scripts with packages via
constructor
.There have been improvements on Continuum’s side that should help this. In short, repodata (the package index for a given channel) was being generated for each anaconda.org query. This was unnecessarily high cost, and some caching schemes have been implemented.
Handling removal of unpinned/improperly pinned packages.
Has been done manually thus far.
This doesn’t scale well though.
Should we (semi) automate removal?
Should we hot-fix broken packages? ( conda forge/conda forge.github.io#170 )
Not currently buildable packages
In particular open source code that is out of scope for CIs.
Examples include Qt4, Qt5, possibly PyQt4, possibly PyQt5, gcc, VTK, etc.
How do we indicate they are built manually?
Are we ok with uploading non-built binaries?
When do we determine something is ok to be built manually?
What procedures should people follow for building manually?
* Use a standard build docker image, VM, or vagrant file
Sign package?
Implement reproducible builds where feasible (linux)
* [](https://reproducible-builds.org/)[https://reproducible-builds.org/](https://reproducible-builds.org/)
What changes do we need to make in conda-smithy elsewhere?
What other build infrastructure could we utilize?
* Would be nice to provide some volunteer builder abstraction, so that we could have an elastic worker farm that would be somewhat resilient.
Standardizing build images is probably (relatively) easy - how to orchestrate, though?
Conda RPMs: https://github.com/pelson/conda-rpms