2020-04-15 conda-forge core meeting

Attendees

* CJ Wright
* Jonathan Helmus
* Eric Dill
* Matthew Becker
* Cheng Lee
* Marius van Niekerk
* Amy Williams
* Timothy Snyder
* John Kirkham
* Michael Sarahan
* Kale Franz
* Marcel Bargull
* Patrick Sodré
* Uwe Korn
* Sylvain Corlay
* Nehal Wani
* Wolf Vollprecht
* Anthony Scopatz

Agenda

Your agenda items

  • intros for people on the line

    • Tim S.

    • Nehal W.

    • Sylvain

    • Cheng Lee

      • primary day-to-day rep between Anaconda Distro <> CF

  • cfep-12 Removing packages that violate the terms of the source package

    • Vote ended on 2020-03-11. What were the results of the vote?

    • Did we hear back from NumFOCUS, asks Isuru a week ago (Filipe: better yet, who is out NF point of contact? I volunteered but got exactly zero response from the core.)

  • [MRB] CFEP-13

    • appveyor removal

      • With some great work from @isuruf we were able to move isl.

      • The only thing left is libssh2 which works and is blocked on a separate issue.

      • I did a census of everything else (~400 packages). It breaks down into 5 cases

        1. package has a build error and uses appveyor (~10ish)

        2. error in feedstock caused my code to say it had appveyor but didn’t (~20ish)

        3. package has an old branch that was for a PR (should not be there) and uses appveyor, but not on master

        4. package has an old version branch that uses appveyor but not on master

        5. package uses appveyor on master but has not been built in at least 1.5 years

      • Given the above, I think removal of appveyor is completely fine.

    • endpoints on web services bot are done

    • next steps are

        1. enabling token rotations in smithy

        1. enabling the use of the endpoints in the ci setup and smithy

        1. several large-scale migrations to move feedstocks to new system

        1. PR to staged recipes to turn it on for everything

    • Only issue on moving from AppVeyor to Azure is that Python 3.6.x and Python 3.7.x use Profile Guided Optimization (PGO), but Python 3.8 does not use it. If we move to Azure, our Python would be slower. PGO makes Python 30-40% faster.

      • There is a pending PR by jhelmus on moving Python 3.6 to azure (disabling PGO). Appveyor times out.

  • [CJ] Need better (more active?) handling of pinning PRs

    • Now that the bot proposes them it would be good to be more proactive about merging/rejecting/deffering migrations

    • merge them all. godspeed.

    • https://github.com/regro/cf-scripts/issues/962

  • [CJ] Free pycharm/jetbrains licenses for those on core (ping CJ if you need one or follow the link in keybase)

    • In about a year CJ will need to ping JetBrains to renew

  • [MRB] bot census!

    • Please reply to our note with any information you have!

    • from the note, we are organizing around keybase + authy for securing credentials + MFA

  • [Wolf+SC] mirrors

    • hopefully Wolf and Sylvain can put together a CFEP for this

    • OVHCloud interested in providing free hosting and compute to conda-forge. Note: they already host a good portion of mybinder’s trafic and nbviewer. Note: they already offered to host the meta channel thingy by Scopatz.

    • Having access to a non-CDN mirror may be good in the short term when updating a large number of inter-dependent packages. E.g. ROS distributions. This may accelerate migrations.

    • This could also be collocated to some of the build resources.

    • Mamba / QuantStack: Testing automatically looking for mirrors and finding the fastest one

    • conda-web.anaconda.org bypasses the conda.anaconda.org CDN (bandwidth from conda-web costs Anaconda more, does not have repodata patches)

    • Anaconda maintains repodata metadata patching, need to figure out how to manage a mirror of that

    • If you’re going to mirror then you should mirror the CDN repodata.json

    • https://pypi.org/project/conda-mirror/ https://github.com/regro/conda-mirror

    • How do you get an account on OVHCloud?

    • Should conda-forge think about moving to a model where we upload from CI -> blob storage -> anaconda.org? Then conda-forge can maintain its own backup.

    • Security considerations around MITM.

    • Schedule follow-on discussion

      • Sylvain (sylvain.corlay@quantstack.net)

      • Wolf

      • Marius

      • Kirkham

      • Uwe

      • Patrick

      • Eric

      • Matthew Becker (becker.mr@gmail.com)

      • Jonathan (check internally for who else)

      • Cheng (clee@anaconda.com)

      • Scopatz (scopatz@gmail.com)

      • Kale (kfranz@anaconda.com)

    • (Nehal) Is conda going to support multiple mirrors for a given channel? (Try another mirror if one times out/is slow)

      • Mamba is - although in the case of the public channels, it may not make sense because there is a CDN.

      • [Kale] I’d say definitely something to consider for Conda. But signing/artifact verification needs to come into that consideration. That’s something we’re actively working on now though, so now is a good time to bring it into the conversation.

  • [Uwe] conda-build issues

    • Currently we have a lot of the latest versions marked as broken

    • Mostly prefix replacements

    • Jonathan to respond on the issue

  • [Kale] Proposal for a Working Group dedicated to major Conda features and design initiatives that broadly affect the ecosystem as a whole

    • top of list stuff: pypy, namespaces

    • weekly-ish cadence, can shift as needed

    • (eric) would love an update / read-out occasionally at the core meeting.

    • if interested, add your name here:

      • Marius

      • Scopatz

      • Matthew Becker (becker.mr@gmail.com)

      • Sylvain

      • Wolf

      • Cheng

      • Eric (maybe)

      • Michael Sarahan

      • Nehal (Would like to see https://pastebin.com/raw/2bFFM76u implemented someday!)

      • Marcel (hopefully starting June – if Bioconda-Job things work out)

      • Kirkham

      • Filipe Fernandes (ocefpaf@gmail.com)

    • [Kale] @Nehal with respect to package signing, we have an engineer 100% dedicated to designing and developing that now. The engineer is Sebastian Awwad https://www.linkedin.com/in/sebastienawwad/ who worked in the TUF group at NYU for a number of years.

    • How does this relate to opening up merge access to conda repos?

      • What do you hope to gain as a maintainer of conda? Some thoughts:

        • Being part of the discussions around direction of conda

        • Taking some pressure of the distro team

  • [CJ] Institutional Partners page in docs

    • TODO: Submit skeleton for PR into conda-forge.github.io repo

Active votes

Subteam updates

Bot

  • [CJ/Matt] Prep for R 4.0.0 migration underway

    • see status here: https://github.com/conda-forge/conda-forge.github.io/issues/1025

    • Use Mamba to check solvability before PR (for migrations that can do that) so that less PRs are opened where builds will knowingly fail, enabling automerge to be more succesful

    • Made PR to add automerge to all r-* feedstocks w/ conda-forge/r on the team, using cran_mirror, and not r-base. (in agreement with the conda-forge/r team)

  • [CJ] Better understanding/handling of run_exports

    • Particularly applicable to issues involving boost and header only deps https://github.com/regro/cf-scripts/issues/960

ARM

POWER

CUDA

Docs

staged-recipes

website

security+systems

  • see bot census above

CI infrastructure

Compiler upgrade

CFEP updates

Open PRs

  • cfep-04 X11 and CDT policy

    • INACTIVE - Merge in with some inactive-esque status?

    • Needs new champion. Thanks for your work on this pkgw! Has unaddressed comments from pkgw as from Jan 10, 2020

  • cfep-06 Staged-recipes review lifecycle

    • INACTIVE - Merge in with some inactive-esque status?

    • Lingering comment from @saraedum. @jakirkham, can you reply? Has unadressed comment from @saraedum from Jan 8, 2020

  • cfep-10 Feedstock statuses, unmaintained

    • INACTIVE - Merge in with some inactive-esque status?

    • Needs another review. Has unaddressed updates from pkgw as of Jan 11, 2020

  • cfep-12 Removing packages that violate the terms of the source package

    • Active debate about moving to “broken” vs deleting from conda-forge channel

    • Active vote, ends on 2020-03-11

    • What were the results of the vote?

    • Did we hear back from NumFOCUS?

Discussion

Check in on previous action items

Copy previous action items from last meeting agenda.

Last meeting

  • [ ] (Dill) schedule mirroring conversation

  • [ ] (Kale) schedule conda working group

  • [ ] (CJ) Merge all the pinnings PRs

  • [ ] (MRB) CFEP-13 next steps?

    • [ ] enabling token rotations in smithy

    • [ ] enabling the use of the endpoints in the ci setup and smithy

    • [ ] several large-scale migrations to move feedstocks to new system

    • [ ] PR to staged recipes to turn it on for everything

  • [ ] (CJ) Institutional Partners page in docs

    • [ ] TODO: Submit skeleton for PR into conda-forge.github.io repo

2 meetings ago

3 meetings ago

  • [x] (Matt Becker) Update the docs with our current thinking / principles.

Move to Issue Tracker

  • [ ] cfep-10 next steps: CJ to call a vote for feedback

  • [ ] cfep-06 next steps: Ask staged recipes team to champion this CFEP and move it forward

  • [ ] jakirkham & CJ-wright to sync on adding CUDA to the migration bot

  • [ ] (Eric) Scheduling Anaconda <-> conda-forge sync on anaconda.org requirements gathering

    • Will try and get this scheduled in the next month.

  • [ ] (Anthony) Reach out to NumFocus to figure out legal ramifications of not including licenses in files.

  • [ ] (Eric) check internally for funding levels for hotels & flying folks from the community in?

  • [ ] (Eric) Figure out finances of conda-forge to support themselves?

  • [ ] (jjhelmus) Open up CFEP for which python’s we’re going to support

  • Remove conda forge readthedocs.

  • [ ] (jakirkham) write a blog post on CUDA stuff we discussed today

  • [ ] (jakirkham) update docs on how to add CUDA support to feedstocks

  • [ ] (jakirkham) will open an issue on conda-smithy to investigate Drone issues. (ping the aarch team)

    • https://github.com/conda-forge/conda-forge.github.io/issues/954