2020-11-18 conda-forge core meeting

Zoom link What time is the meeting in my time zone last weeks meeting

Attendees

  • Filipe Fernandes

  • CJ

  • Nikolay

  • Eric Dill

  • Cheng Lee

  • Connor Martin

  • John Kirkham

  • Ray Douglas

  • Chris Burr

  • Marcel Bargull

  • Paul Ivanov

  • Isuru

  • Michael Sarahan

  • Crystal Soja

  • Wolf

  • Sylvain Corlay

Agenda

Standing items

  • [x] intros for new folks on the call

    • Nikolay from intel

  • [x] (CJ) budget

    • current approvals?

    • Whenever updated numbers land, please screenshare and show the budget.

      • Link is in Keybase (numfocus_spreadsheets.txt)

    • (CJ) We’re all up to date and Oct P&L hasn’t been published yet

  • [x] open votes

  • [ ] (MRB/ED/SC) Roadmap / Funding

    • goal is to spend 15 minutes each core meeting for ~3-4 meetings to discuss this

      • Save last 15 minutes for this.

    • https://hackmd.io/0zGSUS71SbOdBsdLtDmGjg

    • notes will get added to hackmd above

    • MRB will collate into a document of sorts

    • some resources

      1. Some numbers:

        • https://github.com/conda-forge/by-the-numbers/blob/master/conda-forge-timelines.ipynb

        • conda-forge has added about 3k feedstocks per year in 2019 and will in 2020

        • the growth in the amount of data we store appears to be accelerating

      2. risk measurements

        • CJ deserves all of the credit for this idea

        • https://docs.google.com/spreadsheets/d/1ADNNauwVZWUsEdlh5aEg0OLjyDWvCX7PLoo-K34EqcM/edit#gid=0

    • going to skip today due to my own constraints

    • TODO:

      • Everyone take a look at the pypa roadmap:

      • fill out the risk measurement spread sheet: https://github.com/psf/fundable-packaging-improvements/blob/master/FUNDABLES.md

From previous meeting(s)

  • [x] (MB) asking core members to move to “emeritus” status

    • TODO: Eric to set up quarterly check-in for all core members to see if they’re interested in remaining “active” or if they want to move to emeritus

      • Remove emeritus folks from having access to various credentials (api tokens, twitter password, etc.)? This would require a change to the governance doc.

  • [x] (CJ) We seem to have some artifacts that have more files than expected

    • ~4~ 5 pkgs ship requests (other than requests itself) see below

    • What should we do about these?

      • Old pkgs

      • Newer/latest pkgs

    • Track on issue here: https://github.com/conda-forge/conda-forge.github.io/issues/1189

'apache-libcloud-2.4.0-py27_1000',
'apache-libcloud-2.4.0-py36_1000',
'apache-libcloud-2.4.0-py37_1000',
'apache-libcloud-2.4.0-py_1000',
'autovizwidget-0.12.6-py_1000',
'dbxfs-1.0.38-py_0',
'google-api-core-1.20.0-py36h9f0ad1d_0',
'google-api-core-1.20.0-py37hc8dfbb8_0',
'google-api-core-1.20.0-py38h32f6830_0',
'google-api-core-1.20.1-py36h9f0ad1d_0',
'google-api-core-1.20.1-py37hc8dfbb8_0',
'google-api-core-1.20.1-py38h32f6830_0',
'google-cloud-bigquery-storage-core-2.0.0-pyh9f0ad1d_1',
'google-cloud-bigquery-storage-core-2.0.1-pyh9f0ad1d_0'

Your new() agenda items

  • [x] (IF/MRB/MV) intel oneAPI

    • TL;DR it appears we are free to ship the oneAPI compiler stack from intel

    • See these issues:

      • https://github.com/conda-forge/staged-recipes/pull/12527#pullrequestreview-531443464

      • https://github.com/conda-forge/intel_repack-feedstock/issues/16

    • License issues

      • can we ship software other than ISSL?

    • Place to raise issues

      • eg: MKL 2020.4 is segfaulting on an SVD

      • eg: TBB 2020.4 is not available for win/osx because of a hang

    • rough plan

      • figure out what exactly we’d like to ship

        • full intel compiler stack on conda-forge?

        • repackage intel provided conda-packages of the compilers?

        • intelmpi

        • opencl_rt

        • dpcpp_rt

        • oneCCL

      • do the thing

        • implement archspec stuff so we can turn on higher level SIMD optimization levels

          • plan to make work for gcc and clang too

        • allow people to optionally add intel stuff to their feedstocks via migrator (like aarch/ppc and osx arm) or admin command

          • (IF): Why would we need a migrator?

          • (MRB): make things nice for people

          • (MRB): solve any abi issues around fortran or other runtimes?

          • (IF): This would be a huge rebuild with little to no benefit for 99.9% of packages

          • (MRB): OK - happy to use an admin command to add it to feedstocks as needed

    • Other notes

      • https://github.com/jeffhammond/dpcpp-tutorial

    • todo

      • [ ] (Nikolay) licensing for opencl_rt

      • [X] (Nikolay) intelmpi ABI compat w/ mpich * Yes, Intel MPI have ABI compatibility and have plans to support it going forward

      • [ ] (MRB/IF) figure out how exactly to package C/C++ compilers

      • [ ] (MRB/IF) think about fortran ABI

      • [x] (MRB) make conda-forge compilers room (add people including keith)

  • [x] (CJ) Diversity and Inclusion Sub-team

    • https://github.com/conda-forge/conda-forge.github.io/pull/1187

  • [x] (CJ) Please fill out the critical infrastructure survey

    • https://docs.google.com/spreadsheets/d/1ADNNauwVZWUsEdlh5aEg0OLjyDWvCX7PLoo-K34EqcM/edit#gid=0

    • results so far: https://nbviewer.jupyter.org/gist/beckermr/9c0f5aa71720cf1b18646ccd0c3ab40f

  • [x] (MRB) bot now supports PRs into branches!

    • (UK) Thanks!

  • [x] (SC) OVH is giving us s3 storage that should be enough for storing all of CF alongside the VM

    • MSFT would be able to give CF credits for hosting ~$10k

Pushed to next meeting

  • [ ] (MRB/IF) pybind11 packaging

    • issue: https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/849#issuecomment-727207060

    • we agreed to a pybind11-abi metapackage that

      • is versioned with the pybind11 internal abi

      • has a run export on itself

      • pybind11 will have a run_constrained on its version

      • can be optionally added to host envs by users to enforce ABI compat as needed

Active votes

Subteam updates

Bot

ARM

POWER

CUDA

Docs

staged-recipes

website

security+systems

CI infrastructure

Compiler upgrade

CFEP updates

Open PRs

  • cfep-04 X11 and CDT policy

    • INACTIVE - Merge in with some inactive-esque status?

    • Needs new champion. Thanks for your work on this pkgw! Has unaddressed comments from pkgw as from Jan 10, 2020

  • cfep-06 Staged-recipes review lifecycle

    • INACTIVE - Merge in with some inactive-esque status?

    • Lingering comment from @saraedum. @jakirkham, can you reply? Has unadressed comment from @saraedum from Jan 8, 2020

    • (MRB) The stalebot has solved the worst of the issues here. I think we could defer this one permanently.

  • cfep-10 Feedstock statuses, unmaintained

    • INACTIVE - Merge in with some inactive-esque status?

    • Needs another review. Has unaddressed updates from pkgw as of Jan 11, 2020

  • cfep-12 Removing packages that violate the terms of the source package

    • Stalled since May 26, 2020

    • Active debate about moving to “broken” vs deleting from conda-forge channel

    • Active vote, ends on 2020-03-11

    • What were the results of the vote?

    • Did we hear back from NumFOCUS?

  • cfep-17 Handling pin backports and dependency rebuilds

    • Stalled debate about implementation details between Isuru, CJ and Matt

    • UPDATE 2020-07-22: We in principle have agreement to render the extra pinnings needed directly in the feedstock on a temporary basis (i.e., until the migration has ended).

  • cfep-20

Discussion

Check in on previous action items

Copy previous action items from last meeting agenda.

This meeting

2020-11-18

  • [ ] (IF/MRB/MV) intel oneAPI

    • todo

      • [ ] (Nikolay) licensing for opencl_rt

      • [ ] (Nikolay) intelmpi ABI compat w/ mpich

      • [ ] (MRB/IF) figure out how exactly to package C/C++ compilers

      • [ ] (MRB/IF) think about fortran ABI

      • [ ] (MRB) make conda-forge compilers room (add people including keith)

  • [ ] (MB) asking core members to move to “emeritus” status

    • [ ] TODO: Eric to set up quarterly check-in for all core members to see if they’re interested in remaining “active” or if they want to move to emeritus

      • Remove emeritus folks from having access to various credentials (api tokens, twitter password, etc.)? This would require a change to the governance doc.

Last meeting

2020-11-11

  • TODO: Think about bringing in JOSS to provide context around how we might best write papers

2 meetings ago

2020-11-03

  • [x] (MRB) proposed policy on when core pushes to the feedstocks they don’t maintain * [x] (MRB) put in docs PR * [ ] (MRB) make PR on bot to mention the policy

  • TODO: Check on Forrest Watters permissions for core

  • [x] (FF) Outreachy would cost 6500 USD.

    • Next steps: write abstract and vote on spending of funds.

Move to Issue Tracker

2020-10-28 2020-10-21

  • [ ] (Marius?) Python 2.7 migration

    • ( ) [ ] make a hint

    • ( ) [ ] make an announcement

    • ( ) [ ] make the hint a lint

2020-10-07

  • [ ] Make sure to add the NVBug info to the cudatoolkit package that conda-forge makes (if we make one)

2020-09-30

2020-09-23

  • [x] (MRB)

    • do libgfortran name change

    • add target platform to hashes

    • do gfortran migration with bot

    • bump pinnings

2020-09-16

  • [x] Get a call set up with Jon Mease about the kaleido staged recipes PR

    • Emailed on 2020-09-16

  • [x] (FF) Open up a PR on the python feedstock for python 3.9 and see what fails

2020-09-09

  • [ ] (ED) Update governance docs with similar voting model as what got put into conda-tools (+3 with no -1 is a pass)

  • [ ] (SC) Write jinja template to turn institutional partners yaml into a website https://github.com/conda-forge/conda-forge.github.io/blob/master/src/inst_partners.yaml

  • [ ] (SC) Document what needs to be done to create an OVH account and get access

2020-08-26 Docker hub

  • [ ] (JK) Check in on Azure build workers to see if they have the docker hub limitation.

  • [ ] (JK) work with dockerhub to see if we can get OSS status

    • [ ] Check in again at some point. We haven’t heard back as of 2020-09-23

  • [x] (MRB) start pushing images to quay (https://github.com/conda-forge/docker-images/pull/152)

OVH

  • [ ] (???) build webpage to credit them (and others)

  • [ ] If we’re adding a logo, will want to make sure that we have permission to use it.

  • [ ] Shout-out on twitter at some point. “Thanks forOVHCloud for providing a VM”, etc. (maybe after we ship qt on windows with it?)

  • [ ] Figure out how to communicate breaking changes to users. Likely should open up an issue immediately for futher discussion. Ping @kkraus, plus capture notes from further up in these meeting notes

  • [ ] John K. will update the cuda toolkit feedstock on the git repo to note the NVBug link to the internal NVIDIA issue tracker

  • [ ] Jonathan will update docs to note that some non-exhaustive list of packages (like cuda-toolkit, MKL, etc.)

  • [ ] Jonathan will review this PR

  • [ ] (Kale) schedule conda working group

  • [ ] cfep-10 next steps: CJ to call a vote for feedback

  • [ ] cfep-06 next steps: Ask staged recipes team to champion this CFEP and move it forward

  • [ ] jakirkham & CJ-wright to sync on adding CUDA to the migration bot

  • [ ] (Eric) Scheduling Anaconda <-> conda-forge sync on anaconda.org requirements gathering

    • Will try and get this scheduled in the next month.

  • [ ] (Anthony) Reach out to NumFocus to figure out legal ramifications of not including licenses in files.

  • [ ] (Eric) check internally for funding levels for hotels & flying folks from the community in?

  • [ ] (Eric) Figure out finances of conda-forge to support themselves?

  • [ ] (jjhelmus) Open up CFEP for which python’s we’re going to support

  • [ ] (jakirkham) write a blog post on CUDA stuff we discussed today

  • [ ] (jakirkham) update docs on how to add CUDA support to feedstocks

  • [ ] (jakirkham) will open an issue on conda-smithy to investigate Drone issues. (ping the aarch team)

    • https://github.com/conda-forge/conda-forge.github.io/issues/954

  • [ ] (ED) Who we are page? Some combination of a FAQ and a who is everyone. FAQ things like:

    • who’s the POC for CF <> Anaconda, CF <> NumFocus, CF <> Azure

    • who’s the POC for the various subteams?

    • Informal information: roles, day jobs, bios, the whole nine yards, why you’re here, etc.

    • Public or internal? I don’t really care either way. Anyone feel strongly one way or the other?

    • opt-in to public bios

    • software carpentry has a large number of instructors and has https://carpentries.org/instructors

    • some concern about “yet another place to keep stuff up to date”

  • [ ] (ED) document strategies for reproducible environments using conda-forge

  • [ ] (UK) Static libraries stuff

    • [ ] Add linting hints to builds to find them

    • [x] Recommend how to package them -> CFEP-18

    • [x] We should write docs saying we don’t provide support and this is a bad idea. -> CFEP-18