2020-10-21 conda-forge core meeting¶
Zoom link What time is the meeting in my time zone last weeks meeting
Attendees¶
* Eric D
* Matt B
* Filipe
* Cheng Lee
* Isuru
* Marius
* Crystal
* Marcelo
* Paul I.
* Marcel B.
* Connor M.
* Bill Batey
* Stephanie
* John K.
* Lori B.
* Wolf
* CJ
Agenda¶
Standing items¶
[x] intros for new folks on the call
Bill - new to Anaconda. On the distribution team with Cheng and Connor.
[x] (CJ) budget
current approvals?
Whenever updated numbers land, please screenshare and show the budget.
Link is in Keybase (numfocus_spreadsheets.txt)
[x] (MRB) open votes
Chris Burr for Core
Won’t use polys again because something totally went sideways. Maybe we’ll try again?
From previous meeting(s)¶
[x] (MRB) github releases and conda-forge-sparta
code and data in the regro github org right now
if you want access, LMK
I am also happy to move the repos to the conda-forge org but they may slow rerendering since they use a fair bit of github actions
how it works
all packages that are validated and copied to anaconda.org/conda-forge are also sent to a releases repo via a workflow dispatch event in github actions
the github actions CI job (regro/releases)
downloads the package
computes its repodata information
uploads the package and its repodata as a github release
pushes the repodata shard to another repo with all of the shards
another job on a continuous loop (regro/repodata aka the repoworker)
downloads the shards
figures out which ones are new
applies repodata patching to the shards (will also repatch everything if patches have changed)
computes the repodata and channel data
computes an index of links that point to where one can download the package
uploads all of this data as a github release
kicks the heroku server to download the new links
the herkou server (https://conda-forge-sparta.herokuapp.com/conda-forge-sparta/)
gets a request for a package or repodata file at the standard URL for an anaconda server
looks up where that package is in the links index
issues a 300 to redirect the request to the actual location
a final set of cron jobs runs once an hour to sync our repodata shards with anaconda.org (regro/repodata-shards)
this job catches any missed releases due to bad CI jobs or manual uploads
will update the shards repo and upload to github
ensures eventual consistency on ~1-2 hour time scales if things break
overall time delay from the initial workflow dispatch to a package being available is 5-10 minutes
can go longer if repodata patches are being redone for everything
how could this not depend on anaconda if we needed it
replace cf-staging and anaconda.org w/ quetz
limit traffic to those instances to keep the costs down
send the vast majority of download requests to our (possibly beefed up) heroku server and thus github
I really don’t want to maintain this for production, but we could if we had to.
For now, we have a system to help index and archive our data in another spot.
[ ] (MRB / ED / SC) Roadmap / Funding? Discussion in gitter:
(SC) What would CF do as a project if it received funding? Maybe having some kind of whitepapers on the conda-forge website would help. Would conda-forge do something very different if some corporation gave 10k, 100k, or 1M to the project via NumFOCUS?
10k: does it buy enough hosting to do something interesting?
100k - what does it bring.
A much larger amount: conda-forge may even be able to have funded people directly working on the project.
We can almost certainly find a lot useful stuff that would not hurt getting funding for without necessarily having to get consensus on a roadmap. (something i think would not be controversial, like oh we could use some funding to do some nice web development for better visualization of migrations, or stuff) there are probably a lor of things of that nature.
If we make a list, Santa may just show up…
see the bloomberg note above…
wishlist and ideas: https://hackmd.io/0zGSUS71SbOdBsdLtDmGjg
(ED) Can we add initials for whomever submitted the idea? would be useful to track that if there are follow-on questions
(MRB) Start chatting on this for 15 minutes in the next core meeting
[x] (MvN) tactical things to do about cython?
CPython breaks its apis reguarly enough to force a recythoning of much of the python ecosystem.
Should we patch our cython so that when the
CONDA_BUILD
env var is set, we always recythonize and ignore mtime for files?
[x] (MvN) noarch: python v2
As we increasingly drop compatibility with py27/35/36 should we automatically add python constraints to all
noarch: python
artifacts using repodata pacthing?The idea would be to pick an epoch date (say 2020-10-01) and have the artifacts built after that date get a repodata patch adding a python>=3.6 constraint iff it only had a python constraint.
Additionally we may as well upate the noarch adding bot action that adds noarch to strip all lines containing py27/34/35 selectors from the recipe.
This should allow us to move lots of feedstocks that currently depend on some things only for py27 compat to noarch: python cutting down on the growth rate of artifacts.
to do
( ) [ ] make a hint
( ) [ ] make an announcement
( ) [ ] make the hint a lint
[x] (WV) multisheller github.com/wolfv/multisheller
Your new() agenda items¶
[x] (FF) Outreachy, round 2
[X] (FF) “Feature parity” between feedstock and staged-recipes. For example: CFEP-05 does not contemplate a dev release on staged recipes when there is no stable release to start a feedstock. https://github.com/conda-forge/cfep/blob/master/cfep-05.md
No need for a conda-forge.yml, one can add a conda config file
The broader issue still is not solved, we cannot do cos7
staged recipes needs some work!
Pushed to next meeting¶
Active votes¶
Subteam updates¶
Bot¶
ARM¶
POWER¶
CUDA¶
Docs¶
staged-recipes¶
website¶
security+systems¶
CI infrastructure¶
Compiler upgrade¶
CFEP updates¶
Open PRs¶
cfep-04 X11 and CDT policy
INACTIVE - Merge in with some inactive-esque status?
Needs new champion. Thanks for your work on this pkgw! Has unaddressed comments from pkgw as from Jan 10, 2020
cfep-06 Staged-recipes review lifecycle
INACTIVE - Merge in with some inactive-esque status?
Lingering comment from @saraedum. @jakirkham, can you reply? Has unadressed comment from @saraedum from Jan 8, 2020
(MRB) The stalebot has solved the worst of the issues here. I think we could defer this one permanently.
cfep-10 Feedstock statuses, unmaintained
INACTIVE - Merge in with some inactive-esque status?
Needs another review. Has unaddressed updates from pkgw as of Jan 11, 2020
cfep-12 Removing packages that violate the terms of the source package
Stalled since May 26, 2020
Active debate about moving to “broken” vs deleting from conda-forge channel
Active vote, ends on 2020-03-11
What were the results of the vote?
Did we hear back from NumFOCUS?
cfep-17 Handling pin backports and dependency rebuilds
Stalled debate about implementation details between Isuru, CJ and Matt
UPDATE 2020-07-22: We in principle have agreement to render the extra pinnings needed directly in the feedstock on a temporary basis (i.e., until the migration has ended).
Discussion¶
Check in on previous action items¶
Copy previous action items from last meeting agenda.
This meeting¶
2020-10-21
[ ] Python 2.7 migration
( ) [ ] make a hint
( ) [ ] make an announcement
( ) [ ] make the hint a lint
Last meeting¶
2020-10-07
[ ] Make sure to add the NVBug info to the cudatoolkit package that conda-forge makes (if we make one)
2 meetings ago¶
2020-09-30
Move to Issue Tracker¶
2020-09-23
[x] (MRB)
do libgfortran name change
add target platform to hashes
do gfortran migration with bot
bump pinnings
2020-09-16
[x] Get a call set up with Jon Mease about the kaleido staged recipes PR
Emailed on 2020-09-16
[x] (FF) Open up a PR on the python feedstock for python 3.9 and see what fails
2020-09-09
[ ] (ED) Update governance docs with similar voting model as what got put into conda-tools (+3 with no -1 is a pass)
[ ] (SC) Write jinja template to turn institutional partners yaml into a website https://github.com/conda-forge/conda-forge.github.io/blob/master/src/inst_partners.yaml
[ ] (SC) Document what needs to be done to create an OVH account and get access
2020-08-26 Docker hub
[ ] (JK) Check in on Azure build workers to see if they have the docker hub limitation.
[ ] (JK) work with dockerhub to see if we can get OSS status
[ ] Check in again at some point. We haven’t heard back as of 2020-09-23
[x] (MRB) start pushing images to quay (https://github.com/conda-forge/docker-images/pull/152)
OVH
[ ] (???) build webpage to credit them (and others)
[ ] If we’re adding a logo, will want to make sure that we have permission to use it.
[ ] Shout-out on twitter at some point. “Thanks forOVHCloud for providing a VM”, etc. (maybe after we ship qt on windows with it?)
[ ] Figure out how to communicate breaking changes to users. Likely should open up an issue immediately for futher discussion. Ping @kkraus, plus capture notes from further up in these meeting notes
[ ] John K. will update the cuda toolkit feedstock on the git repo to note the NVBug link to the internal NVIDIA issue tracker
[ ] Jonathan will update docs to note that some non-exhaustive list of packages (like cuda-toolkit, MKL, etc.)
[ ] Jonathan will review this PR
[ ] (Kale) schedule conda working group
[ ] cfep-10 next steps: CJ to call a vote for feedback
[ ] cfep-06 next steps: Ask staged recipes team to champion this CFEP and move it forward
[ ] jakirkham & CJ-wright to sync on adding CUDA to the migration bot
[ ] (Eric) Scheduling Anaconda <-> conda-forge sync on anaconda.org requirements gathering
Will try and get this scheduled in the next month.
[ ] (Anthony) Reach out to NumFocus to figure out legal ramifications of not including licenses in files.
[ ] (Eric) check internally for funding levels for hotels & flying folks from the community in?
[ ] (Eric) Figure out finances of conda-forge to support themselves?
[ ] (jjhelmus) Open up CFEP for which python’s we’re going to support
[ ] (jakirkham) write a blog post on CUDA stuff we discussed today
[ ] (jakirkham) update docs on how to add CUDA support to feedstocks
[ ] (jakirkham) will open an issue on conda-smithy to investigate Drone issues. (ping the aarch team)
https://github.com/conda-forge/conda-forge.github.io/issues/954
[ ] (ED) Who we are page? Some combination of a FAQ and a who is everyone. FAQ things like:
who’s the POC for CF <> Anaconda, CF <> NumFocus, CF <> Azure
who’s the POC for the various subteams?
Informal information: roles, day jobs, bios, the whole nine yards, why you’re here, etc.
Public or internal? I don’t really care either way. Anyone feel strongly one way or the other?
opt-in to public bios
software carpentry has a large number of instructors and has https://carpentries.org/instructors
some concern about “yet another place to keep stuff up to date”
[ ] (ED) document strategies for reproducible environments using conda-forge
[ ] (UK) Static libraries stuff
[ ] Add linting hints to builds to find them
[x] Recommend how to package them -> CFEP-18
[x] We should write docs saying we don’t provide support and this is a bad idea. -> CFEP-18