2016-08-25: General discussion

Time: 14:00 UTC

Hangout link: https://hangouts.google.com/call/v5olhwzpfzgzpoq5i3wthjpqpie

Attendees

Jonathan Helmus, Filipe, Michael Sarahan, John Kirkham, Jake VanderPlas, Eric Dill, Ray Donnelly , Phil Elson

Standing items

  • How many repos? 1035

  • How many contributors? 212 (with a few bots)

  • New core devs?

Notes

  • Invite Peter M. Landwehr (pmlandwehr) to be involved with review of staged-recipes. Should we give these type of people a title, Filipe will reach out to.

  • Governing Open Source Projects at Scale: Lessons from Wikipedia’s Growing Pains | Staurt Geiger https://www.youtube.com/watch?v=ZSQJYEVcMWM&index=89&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6

  • Enhancement proposal document, Jonathan has notes will write these up later today.

  • Governance document - help is welcomed. Also “whos who” or “about” page. https://conda-forge.github.io/#about

    *   This page could be expanded, should mentioned these meeting.

  • Removing items from agenda

    *   Prioritize items on agenda which we should/must talk about.

    • Cross link items to GitHub issues/discussions

  • Status page: https://conda-forge.github.io/status/

    *   Linked to "status" repo:  [](https://github.com/conda-forge/status)[https://github.com/conda-forge/status](https://github.com/conda-forge/status)

  • conda-forge code of conduct - Filipe still workin on

  • Many groups working on new build systems: Filipe, Phil, Continuum

    *   Continuum's  plan would allow others to add build workers, perhaps conda-forge could  use these in addition to the CI services, especially for long builds

    • Organize new meeting to discuss this topic

  • Open sourcing Anaconda Build, should we push to get this released?

    *   Would be helpful to have our own build system rather than being dependent on CI systems.

  • Travis CI can increase time if we reduce concurrency

    *   Can we switch between longer time and concurrency?  How much work is this?

    • Probably not going to take offer at the moment

    • Better to find trusted hardware somewhere

    • Vagrant for OS X builds, can we look into this

  • Security

    *   If user changes name and someone takes old name can be a security issue:  [](https://groups.google.com/forum/#)[https://groups.google.com/forum/#](https://groups.google.com/forum/#%21topic/rustlang-security-announcements/BK_3gbXhSn4)[!topic/rustlang-security-announcements/BK_3gbXhSn4](https://groups.google.com/forum/#%21topic/rustlang-security-announcements/BK_3gbXhSn4)

    • Can be solved by using unique user ID rather than GitHub username

    • Want tokens for Anaconda.org which allow writing to a single package (Phil will push Continuum on this) rather than globally.

  • Metadata unification

    *   Should conda-forge include additional metadata which would make it easier for Continuum to re-use recipes

    • Should this be required or optional?

          *   Required would likely reduce number of contributors

      • Will require time/work to add these to all current packages

      • Add to linter and conda skeleton

        *   Make linter have "warnings" not hard fails

      • Many of these seem redundant, can we re-use existing metadata?

    • License file should likely be required

          *   Legal vs. suggested

Agenda

  • Marking agenda items as done.

  • Share status page. :) Also figure out how to direct notifications effectively.

  • Enhancement proposal document update.

  • conda-forge code of conduct doc: https://docs.google.com/document/d/10dxX0Zse0Rx1HqsxC73Wfsghmy5m8PP8cHuBIOhWKpc/edit

  • Mention Travis-CI offer for more CI time.

  • We could look at increasing your build time to 180 mins, but we may need to decrease your default concurrency from 5 jobs to 3 as you will be using multiple VMs for a long period at a time.

  • Mention/Discuss Travis Oliphant’s comment regarding open sourcing Anaconda Build CI.

  • Security

  • Feedstocks philosophy: Explicit vs implicit / reproducible vs redundant

  • Metadata unification with Continuum - are we OK with adding some fields to about section to match Anaconda standard?

  • Including license file

  • Many recipes don’t include the license file.

  • Almost every license has some terms about making the license available.

  • Should we just start requiring this field.

  • Note some developers are not including the license file either.

  • In some cases it has been a struggle to get them to.

  • CUDA/cuDNN update

  • Improving infrastructure

    *   Better workflows with staged-recipes

        *   Fast finish AppVeyor on merge ( [conda forge/staged recipes#1142](https://github.com/conda-forge/staged-recipes/pull/1142) )
*   Drop Travis CI matrix ( [conda forge/staged recipes#1234](https://github.com/conda-forge/staged-recipes/pull/1234) )
*   Use CircleCI for feedstock generation ( [conda forge/staged recipes#916](https://github.com/conda-forge/staged-recipes/issues/916) )
*   Keeping recipes out of PRs ( [conda forge/staged recipes#942](https://github.com/conda-forge/staged-recipes/issues/942) )
*   Bank work in partial conversion ( [conda forge/staged recipes#915](https://github.com/conda-forge/staged-recipes/issues/915) )

  • Notifications (how do we stay on top of them)

  • MSYS2

    *   Available on defaults - was in conda 4.1.7, but that was pulled.  Coming in 4.1.8.

    • Discussing Ray Donnelly’s work on MSYS2 packages and how we want to use and integrate these into conda-forge.

    • Some use cases to consider OpenBLAS, FFTW, build tools, others?

  • Binary data

    *   Do we include it in recipes?

    • What kinds do we allow if any (e.g. icons)?

    • How do we verify the licensing?

    • How do we verify that they are safe?

  • Dev releases: Where do they happen?

    *   Do we do them at conda-forge?

        *   Maybe add a label.

*   Do we let others do them with a feedstock on their own repo?

    • How do we enforce whatever we decide?

  • Channel mirroring

    *   Can this point be a little bit explained? I thought about this as well and would like to contribute to this point.

        *   Eric Dill has put together a script for copying a package from one channel to another here: [conda forge/conda forge.github.io#134](https://github.com/conda-forge/conda-forge.github.io/pull/134)
*   I have a really, really crude script that copies all of the packages in one channel to another that I just put at: [](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555)[https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555)
*   conda-build-all can copy from one channel to another: `conda build-all --inspect-channels conda-forge --upload-channels astropy some_packge_recipe` will copy the `some_package`  from the channel conda-forge to  astropy if it can, or build it if it  doesn't exist on conda-forge. Discussion  about what the desired  behavior should be has started at: [SciTools/conda build all#46](https://github.com/SciTools/conda-build-all/issues/46)

  • Feedstock history

    *   Is it sacred?

    • Do we rebase/force push?

          *   If so, under what conditions?

      • How do we avoid multiple people doing this simultaneously?

                *   I don't think you can.

*   IMHO,  if it's just one author in staged recipes, sure.  If feedstock, no  force push - only to PRs to feedstock.  If people don't mind merge PRs,  it sure is a lot simpler to not rebase.  I have messed up rebasing a few  times recently... =(

  • Continuum metadata request: can we add these to linter?

    *   example metadata: [](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)[https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)

    • Also, distinguish summary (limit of 77 or 80 chars) from description (unlimited)

    • Anaconda verify: would be nice to meet in the middle, rather than diverge. conda-build may integrate anaconda-verify, would be nice if conda-forge added metadata here.

  • Drop numpy 1.10 and reduce our build matrix. (Numba now works with numpy 1.11.)

  • Signing packages

    *   Should be easy to do. ( [](http://conda.pydata.org/docs/signed-packages.html)[http://conda.pydata.org/docs/signed-packages.html](http://conda.pydata.org/docs/signed-packages.html) )

    • There has been some interest previously.

  • HTTPError: 503 Server Error: Service Unavailable: Back-end server is at capacity for url…

    *   Seems we are regularly running into this issue under normal usage conditions.

    • Had discussed previously caching packages on AppVeyor and trying to reuse those to start.

    • Maybe we need to consider caching on all CIs.

    • Building our own Miniconda-like self-extracting scripts with packages via constructor.

    • There have been improvements on Continuum’s side that should help this. In short, repodata (the package index for a given channel) was being generated for each anaconda.org query. This was unnecessarily high cost, and some caching schemes have been implemented.

  • Handling removal of unpinned/improperly pinned packages.

    *   Has been done manually thus far.

  • Not currently buildable packages

    *   In particular open source code that is out of scope for CIs.

    • Examples include Qt4, Qt5, possibly PyQt4, possibly PyQt5, gcc, VTK, etc.

    • How do we indicate they are built manually?

    • Are we ok with uploading non-built binaries?

    • When do we determine something is ok to be built manually?

    • What procedures should people follow for building manually?

          *   Use a standard build docker image, VM, or vagrant file

      • Sign package?

      • Implement reproducible builds where feasible (linux)

                *   [](https://reproducible-builds.org/)[https://reproducible-builds.org/](https://reproducible-builds.org/)

      • What changes do we need to make in conda-smithy elsewhere?

    • What other build infrastructure could we utilize?

          *   Would  be nice to provide some volunteer builder abstraction, so that we could  have an elastic worker farm that would be somewhat resilient.

      • Standardizing build images is probably (relatively) easy - how to orchestrate, though?

  • Staged Releases

  • Windows BLAS Solutions

    *   Still don't have a BLAS for Windows yet need something.

    • Don’t build a BLAS

          *   NumPy has a small subset of BLAS functionality.

      • Not sure what to do with SciPy (unable to find Windows wheels for them either).

      • Build OpenBLAS with C support only.

        *   Will be pretty slow.

      • Should work on all Pythons.

      • Build OpenBLAS with MinGW compilers.

        *   Works with Python 2.7 and 3.4.

      • Won’t work with Python 3.5?

      • Reuse something like R’s BLAS.

        *   Is there a package for something like this?

      • Will it have the same issues with Python 3.5?

      • ATLAS?