2016-08-25: General discussion

Time: 14:00 UTC

Hangout link: https://hangouts.google.com/call/v5olhwzpfzgzpoq5i3wthjpqpie

Attendees

Jonathan Helmus, Filipe, Michael Sarahan, John Kirkham, Jake VanderPlas, Eric Dill, Ray Donnelly , Phil Elson

Standing items

How many repos? 1035
How many contributors? 212 (with a few bots)
New core devs?

Notes

Invite Peter M. Landwehr (pmlandwehr) to be involved with review of staged-recipes. Should we give these type of people a title, Filipe will reach out to.
Governing Open Source Projects at Scale: Lessons from Wikipedia's Growing Pains | Staurt Geiger https://www.youtube.com/watch?v=ZSQJYEVcMWM&index=89&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6
Enhancement proposal document, Jonathan has notes will write these up later today.
Governance document - help is welcomed. Also "whos who" or "about" page. https://conda-forge.github.io/#about
```
*   This page could be expanded, should mentioned these meeting.
```
Removing items from agenda
```
*   Prioritize items on agenda which we should/must talk about.
```
- Cross link items to GitHub issues/discussions

Status page: https://conda-forge.github.io/status/

*   Linked to "status" repo:  [](https://github.com/conda-forge/status)[https://github.com/conda-forge/status](https://github.com/conda-forge/status)

conda-forge code of conduct - Filipe still workin on

Many groups working on new build systems: Filipe, Phil, Continuum

*   Continuum's  plan would allow others to add build workers, perhaps conda-forge could  use these in addition to the CI services, especially for long builds

Organize new meeting to discuss this topic

Open sourcing Anaconda Build, should we push to get this released?

*   Would be helpful to have our own build system rather than being dependent on CI systems.

Travis CI can increase time if we reduce concurrency
```
*   Can we switch between longer time and concurrency?  How much work is this?
```
- Probably not going to take offer at the moment
- Better to find trusted hardware somewhere
- Vagrant for OS X builds, can we look into this

Security

*   If user changes name and someone takes old name can be a security issue:  [](https://groups.google.com/forum/#)[https://groups.google.com/forum/#](https://groups.google.com/forum/#%21topic/rustlang-security-announcements/BK_3gbXhSn4)[!topic/rustlang-security-announcements/BK_3gbXhSn4](https://groups.google.com/forum/#%21topic/rustlang-security-announcements/BK_3gbXhSn4)

Can be solved by using unique user ID rather than GitHub username
Want tokens for Anaconda.org which allow writing to a single package (Phil will push Continuum on this) rather than globally.

Metadata unification

*   Should conda-forge include additional metadata which would make it easier for Continuum to re-use recipes

Should this be required or optional?
```
    *   Required would likely reduce number of contributors
```
- Will require time/work to add these to all current packages
- Add to linter and conda skeleton
```
*   Make linter have "warnings" not hard fails
```
- Many of these seem redundant, can we re-use existing metadata?
License file should likely be required
```
    *   Legal vs. suggested
```

Agenda

Marking agenda items as done.
Share status page. :) Also figure out how to direct notifications effectively.
Enhancement proposal document update.
conda-forge code of conduct doc: https://docs.google.com/document/d/10dxX0Zse0Rx1HqsxC73Wfsghmy5m8PP8cHuBIOhWKpc/edit
Mention Travis-CI offer for more CI time.
We could look at increasing your build time to 180 mins, but we may need to decrease your default concurrency from 5 jobs to 3 as you will be using multiple VMs for a long period at a time.
Mention/Discuss Travis Oliphant's comment regarding open sourcing Anaconda Build CI.
Security
Feedstocks philosophy: Explicit vs implicit / reproducible vs redundant
Metadata unification with Continuum - are we OK with adding some fields to about section to match Anaconda standard?
Including license file
Many recipes don't include the license file.
Almost every license has some terms about making the license available.
Should we just start requiring this field.
Note some developers are not including the license file either.
In some cases it has been a struggle to get them to.
CUDA/cuDNN update

Improving infrastructure

*   Better workflows with staged-recipes

        *   Fast finish AppVeyor on merge ( [conda forge/staged recipes#1142](https://github.com/conda-forge/staged-recipes/pull/1142) )
*   Drop Travis CI matrix ( [conda forge/staged recipes#1234](https://github.com/conda-forge/staged-recipes/pull/1234) )
*   Use CircleCI for feedstock generation ( [conda forge/staged recipes#916](https://github.com/conda-forge/staged-recipes/issues/916) )
*   Keeping recipes out of PRs ( [conda forge/staged recipes#942](https://github.com/conda-forge/staged-recipes/issues/942) )
*   Bank work in partial conversion ( [conda forge/staged recipes#915](https://github.com/conda-forge/staged-recipes/issues/915) )

Notifications (how do we stay on top of them)
MSYS2
```
*   Available on defaults - was in conda 4.1.7, but that was pulled.  Coming in 4.1.8.
```
- Discussing Ray Donnelly's work on MSYS2 packages and how we want to use and integrate these into conda-forge.
- Some use cases to consider OpenBLAS, FFTW, build tools, others?
Binary data
```
*   Do we include it in recipes?
```
- What kinds do we allow if any (e.g. icons)?
- How do we verify the licensing?
- How do we verify that they are safe?

Dev releases: Where do they happen?

*   Do we do them at conda-forge?

        *   Maybe add a label.

*   Do we let others do them with a feedstock on their own repo?

How do we enforce whatever we decide?

Channel mirroring

*   Can this point be a little bit explained? I thought about this as well and would like to contribute to this point.

        *   Eric Dill has put together a script for copying a package from one channel to another here: [conda forge/conda forge.github.io#134](https://github.com/conda-forge/conda-forge.github.io/pull/134)
*   I have a really, really crude script that copies all of the packages in one channel to another that I just put at: [](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555)[https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555)
*   conda-build-all can copy from one channel to another: `conda build-all --inspect-channels conda-forge --upload-channels astropy some_packge_recipe` will copy the `some_package`  from the channel conda-forge to  astropy if it can, or build it if it  doesn't exist on conda-forge. Discussion  about what the desired  behavior should be has started at: [SciTools/conda build all#46](https://github.com/SciTools/conda-build-all/issues/46)

Feedstock history

*   Is it sacred?

Do we rebase/force push?

    *   If so, under what conditions?

How do we avoid multiple people doing this simultaneously?

        *   I don't think you can.

*   IMHO,  if it's just one author in staged recipes, sure.  If feedstock, no  force push - only to PRs to feedstock.  If people don't mind merge PRs,  it sure is a lot simpler to not rebase.  I have messed up rebasing a few  times recently... =(

Continuum metadata request: can we add these to linter?

*   example metadata: [](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)[https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)

Also, distinguish summary (limit of 77 or 80 chars) from description (unlimited)
Anaconda verify: would be nice to meet in the middle, rather than diverge. conda-build may integrate anaconda-verify, would be nice if conda-forge added metadata here.

Drop numpy 1.10 and reduce our build matrix. (Numba now works with numpy 1.11.)

Signing packages

*   Should be easy to do. ( [](http://conda.pydata.org/docs/signed-packages.html)[http://conda.pydata.org/docs/signed-packages.html](http://conda.pydata.org/docs/signed-packages.html) )

There has been some interest previously.

HTTPError: 503 Server Error: Service Unavailable: Back-end server is at capacity for url...
```
*   Seems we are regularly running into this issue under normal usage conditions.
```
- Had discussed previously caching packages on AppVeyor and trying to reuse those to start.
- Maybe we need to consider caching on all CIs.
- Building our own Miniconda-like self-extracting scripts with packages via constructor.
- There have been improvements on Continuum's side that should help this. In short, repodata (the package index for a given channel) was being generated for each anaconda.org query. This was unnecessarily high cost, and some caching schemes have been implemented.
Handling removal of unpinned/improperly pinned packages.
```
*   Has been done manually thus far.
```
- This doesn't scale well though.
- Should we (semi) automate removal?
- Should we hot-fix broken packages? ( conda forge/conda forge.github.io#170 )
- Should we label them as broken
Not currently buildable packages
```
*   In particular open source code that is out of scope for CIs.
```
- Examples include Qt4, Qt5, possibly PyQt4, possibly PyQt5, gcc, VTK, etc.
- How do we indicate they are built manually?
- Are we ok with uploading non-built binaries?
- When do we determine something is ok to be built manually?
- What procedures should people follow for building manually?
```
    *   Use a standard build docker image, VM, or vagrant file
```
  - Sign package?
  - Implement reproducible builds where feasible (linux)
    * [](https://reproducible-builds.org/)[https://reproducible-builds.org/](https://reproducible-builds.org/)
  - What changes do we need to make in conda-smithy elsewhere?
- What other build infrastructure could we utilize?
```
    *   Would  be nice to provide some volunteer builder abstraction, so that we could  have an elastic worker farm that would be somewhat resilient.
```
  - Standardizing build images is probably (relatively) easy - how to orchestrate, though?
Staged Releases
Windows BLAS Solutions
```
*   Still don't have a BLAS for Windows yet need something.
```
- Don't build a BLAS
```
    *   NumPy has a small subset of BLAS functionality.
```
  - Not sure what to do with SciPy (unable to find Windows wheels for them either).
  - Build OpenBLAS with C support only.
    * Will be pretty slow.
  - Should work on all Pythons.
  - Build OpenBLAS with MinGW compilers.
    * Works with Python 2.7 and 3.4.
  - Won't work with Python 3.5?
  - Reuse something like R's BLAS.
    * Is there a package for something like this?
  - Will it have the same issues with Python 3.5?
  - ATLAS?