2020-04-15 conda-forge core meeting¶
Attendees¶
* CJ Wright
* Jonathan Helmus
* Eric Dill
* Matthew Becker
* Cheng Lee
* Marius van Niekerk
* Amy Williams
* Timothy Snyder
* John Kirkham
* Michael Sarahan
* Kale Franz
* Marcel Bargull
* Patrick Sodré
* Uwe Korn
* Sylvain Corlay
* Nehal Wani
* Wolf Vollprecht
* Anthony Scopatz
Agenda¶
Your agenda items¶
intros for people on the line
Tim S.
Nehal W.
Sylvain
Cheng Lee
primary day-to-day rep between Anaconda Distro <> CF
cfep-12 Removing packages that violate the terms of the source package
Vote ended on 2020-03-11. What were the results of the vote?
Did we hear back from NumFOCUS, asks Isuru a week ago (Filipe: better yet, who is out NF point of contact? I volunteered but got exactly zero response from the core.)
[MRB] CFEP-13
appveyor removal
With some great work from @isuruf we were able to move isl.
The only thing left is libssh2 which works and is blocked on a separate issue.
I did a census of everything else (~400 packages). It breaks down into 5 cases
package has a build error and uses appveyor (~10ish)
error in feedstock caused my code to say it had appveyor but didn’t (~20ish)
package has an old branch that was for a PR (should not be there) and uses appveyor, but not on master
package has an old version branch that uses appveyor but not on master
package uses appveyor on master but has not been built in at least 1.5 years
Given the above, I think removal of appveyor is completely fine.
endpoints on web services bot are done
next steps are
enabling token rotations in smithy
enabling the use of the endpoints in the ci setup and smithy
several large-scale migrations to move feedstocks to new system
PR to staged recipes to turn it on for everything
Only issue on moving from AppVeyor to Azure is that Python 3.6.x and Python 3.7.x use Profile Guided Optimization (PGO), but Python 3.8 does not use it. If we move to Azure, our Python would be slower. PGO makes Python 30-40% faster.
There is a pending PR by jhelmus on moving Python 3.6 to azure (disabling PGO). Appveyor times out.
[CJ] Need better (more active?) handling of pinning PRs
Now that the bot proposes them it would be good to be more proactive about merging/rejecting/deffering migrations
merge them all. godspeed.
https://github.com/regro/cf-scripts/issues/962
[CJ] Free pycharm/jetbrains licenses for those on core (ping CJ if you need one or follow the link in keybase)
In about a year CJ will need to ping JetBrains to renew
[MRB] bot census!
Please reply to our note with any information you have!
from the note, we are organizing around keybase + authy for securing credentials + MFA
[Wolf+SC] mirrors
hopefully Wolf and Sylvain can put together a CFEP for this
OVHCloud interested in providing free hosting and compute to conda-forge. Note: they already host a good portion of mybinder’s trafic and nbviewer. Note: they already offered to host the meta channel thingy by Scopatz.
Having access to a non-CDN mirror may be good in the short term when updating a large number of inter-dependent packages. E.g. ROS distributions. This may accelerate migrations.
This could also be collocated to some of the build resources.
Mamba / QuantStack: Testing automatically looking for mirrors and finding the fastest one
conda-web.anaconda.org bypasses the conda.anaconda.org CDN (bandwidth from conda-web costs Anaconda more, does not have repodata patches)
Anaconda maintains repodata metadata patching, need to figure out how to manage a mirror of that
If you’re going to mirror then you should mirror the CDN repodata.json
https://pypi.org/project/conda-mirror/ https://github.com/regro/conda-mirror
How do you get an account on OVHCloud?
Should conda-forge think about moving to a model where we upload from CI -> blob storage -> anaconda.org? Then conda-forge can maintain its own backup.
Security considerations around MITM.
Schedule follow-on discussion
Sylvain (sylvain.corlay@quantstack.net)
Wolf
Marius
Kirkham
Uwe
Patrick
Eric
Matthew Becker (becker.mr@gmail.com)
Jonathan (check internally for who else)
Cheng (clee@anaconda.com)
Scopatz (scopatz@gmail.com)
Kale (kfranz@anaconda.com)
(Nehal) Is conda going to support multiple mirrors for a given channel? (Try another mirror if one times out/is slow)
Mamba is - although in the case of the public channels, it may not make sense because there is a CDN.
[Kale] I’d say definitely something to consider for Conda. But signing/artifact verification needs to come into that consideration. That’s something we’re actively working on now though, so now is a good time to bring it into the conversation.
[Uwe] conda-build issues
Currently we have a lot of the latest versions marked as broken
Mostly prefix replacements
Jonathan to respond on the issue
[Kale] Proposal for a Working Group dedicated to major Conda features and design initiatives that broadly affect the ecosystem as a whole
top of list stuff: pypy, namespaces
weekly-ish cadence, can shift as needed
(eric) would love an update / read-out occasionally at the core meeting.
if interested, add your name here:
Marius
Scopatz
Matthew Becker (becker.mr@gmail.com)
Sylvain
Wolf
Cheng
Eric (maybe)
Michael Sarahan
Nehal (Would like to see https://pastebin.com/raw/2bFFM76u implemented someday!)
Marcel (hopefully starting June – if Bioconda-Job things work out)
Kirkham
Filipe Fernandes (ocefpaf@gmail.com)
[Kale] @Nehal with respect to package signing, we have an engineer 100% dedicated to designing and developing that now. The engineer is Sebastian Awwad https://www.linkedin.com/in/sebastienawwad/ who worked in the TUF group at NYU for a number of years.
How does this relate to opening up merge access to conda repos?
What do you hope to gain as a maintainer of conda? Some thoughts:
Being part of the discussions around direction of conda
Taking some pressure of the distro team
[CJ] Institutional Partners page in docs
TODO: Submit skeleton for PR into conda-forge.github.io repo
Active votes¶
Subteam updates¶
Bot¶
[CJ/Matt] Prep for R 4.0.0 migration underway
see status here: https://github.com/conda-forge/conda-forge.github.io/issues/1025
Use Mamba to check solvability before PR (for migrations that can do that) so that less PRs are opened where builds will knowingly fail, enabling automerge to be more succesful
Made PR to add automerge to all
r-*
feedstocks w/ conda-forge/r on the team, usingcran_mirror
, and not r-base. (in agreement with the conda-forge/r team)
[CJ] Better understanding/handling of run_exports
Particularly applicable to issues involving boost and header only deps https://github.com/regro/cf-scripts/issues/960
ARM¶
POWER¶
CUDA¶
Docs¶
staged-recipes¶
website¶
security+systems¶
see bot census above
CI infrastructure¶
Compiler upgrade¶
CFEP updates¶
Open PRs¶
cfep-04 X11 and CDT policy
INACTIVE - Merge in with some inactive-esque status?
Needs new champion. Thanks for your work on this pkgw! Has unaddressed comments from pkgw as from Jan 10, 2020
cfep-06 Staged-recipes review lifecycle
INACTIVE - Merge in with some inactive-esque status?
Lingering comment from @saraedum. @jakirkham, can you reply? Has unadressed comment from @saraedum from Jan 8, 2020
cfep-10 Feedstock statuses, unmaintained
INACTIVE - Merge in with some inactive-esque status?
Needs another review. Has unaddressed updates from pkgw as of Jan 11, 2020
cfep-12 Removing packages that violate the terms of the source package
Active debate about moving to “broken” vs deleting from conda-forge channel
Active vote, ends on 2020-03-11
What were the results of the vote?
Did we hear back from NumFOCUS?
Discussion¶
Check in on previous action items¶
Copy previous action items from last meeting agenda.
Last meeting¶
[ ] (Dill) schedule mirroring conversation
[ ] (Kale) schedule conda working group
[ ] (CJ) Merge all the pinnings PRs
[ ] (MRB) CFEP-13 next steps?
[ ] enabling token rotations in smithy
[ ] enabling the use of the endpoints in the ci setup and smithy
[ ] several large-scale migrations to move feedstocks to new system
[ ] PR to staged recipes to turn it on for everything
[ ] (CJ) Institutional Partners page in docs
[ ] TODO: Submit skeleton for PR into conda-forge.github.io repo
2 meetings ago¶
3 meetings ago¶
[x] (Matt Becker) Update the docs with our current thinking / principles.
Move to Issue Tracker¶
[ ] cfep-10 next steps: CJ to call a vote for feedback
[ ] cfep-06 next steps: Ask staged recipes team to champion this CFEP and move it forward
[ ] jakirkham & CJ-wright to sync on adding CUDA to the migration bot
[ ] (Eric) Scheduling Anaconda <-> conda-forge sync on anaconda.org requirements gathering
Will try and get this scheduled in the next month.
[ ] (Anthony) Reach out to NumFocus to figure out legal ramifications of not including licenses in files.
[ ] (Eric) check internally for funding levels for hotels & flying folks from the community in?
[ ] (Eric) Figure out finances of conda-forge to support themselves?
[ ] (jjhelmus) Open up CFEP for which python’s we’re going to support
Remove conda forge readthedocs.
[ ] (jakirkham) write a blog post on CUDA stuff we discussed today
[ ] (jakirkham) update docs on how to add CUDA support to feedstocks
[ ] (jakirkham) will open an issue on conda-smithy to investigate Drone issues. (ping the aarch team)
https://github.com/conda-forge/conda-forge.github.io/issues/954