Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

NumFOCUS SDG 2023 Round 1 Proposal

Name of Submitter:

Jaime Rodríguez-Guerra

Your Email:

[email protected]

Is your project Sponsored or Affiliated?

Fiscally Sponsored

Select Your Project:

conda-forge

Proposal Title:

Access control improvements for opt-in CI (Continuous Integration) services

Two Sentence Summary of Proposal:

To implement an open and transparent mechanism to grant and maintain access control for CI services in conda-forge. Such a mechanism will allow project's maintainers to request on-demand CI services for their packages, in the event the default runner capabilities are not sufficient (e.g. GPU-enabled libraries, build times longer than 6h, insufficient memory or disk space).

Description of Proposal:

Conda-forge relies on 3rd party Continuous Integration (CI) services to build the thousands of packages maintained by its community. Azure Pipelines handles most of the load and is the default CI service for Linux, macOS and Windows. conda-forge does support other CI providers, like Travis, Circle or Drone, for non-x64 Linux architectures (ARM64 and PowerPC). Still, the availability of these runners is much more limited than Azure's, leading to long waiting times and numerous servicing errors that hinder conda-forge's performance.

To alleviate all these problems, conda-forge is considering making some non-default providers available to projects that satisfy certain eligibility criteria (see https://github.com/conda-forge/conda-forge.github.io/issues/1875). Such an approach requires some sort of access control mechanism with the following features:

  • A request protocol where maintainers can apply for usage of certain CI providers, and authorized members can review and approve such requests.
  • A public list of approved projects and the access granted within each resource.
  • A way of revoking access to previously authorized resources if needed.
  • The only manual steps in the process should be the review and approval. Everything else should happen in an automated way.

We propose a mechanism inspired by the procedures followed in https://github.com/conda-forge/admin-requests, with a publicly available list of the allocated resources per project that follows established best practices in the Infrastructure as Code community.

Please explain the benefit of this proposal including:

  • Impact to the project
  • Impact to the scientific ecosystem
  • Impact to the community

Having an opt-in mechanism for specialized CI services in conda-forge will not only make the allocation of scarce resources more fair and transparent, but it will also pave the way for the implementation of previously unavailable building strategies.

For example, GPU-enabled runners were requested a few years ago (see https://github.com/conda-forge/conda-forge.github.io/issues/63, dated 2016), but conda-forge is still unable to provide a secure and fair way to deliver this service. Thanks to the collaboration of several conda-forge partners, a prototype workflow is now available (see https://github.com/conda-forge/cf-autotick-bot-test-package-feedstock/pull/446); however it cannot be made freely available to the community without an access control mechanism.

A second example of such a need is the hypothetical availability of cloud computing resources donated by a generous institution. Ensuring that the donated credits are available to the requested (or best-suited) projects would also need to happen through the same access control mechanism.

Overall, the proposal hereby submitted will significantly impact how conda-forge builds its packages beyond publicly available runners with time-limited resources by:

  1. Enabling secure access to specialized CI services and resources, which will allow conda-forge to operate more reliably by reducing waiting times and service availability errors.
  2. Allowing projects to benefit from new building features without hindering the existing infrastructure's reliability.
  3. Providing a mechanism for donors (or sponsors) to support conda-forge by sponsoring or donating cloud computing credits easily.
  4. Diversifying the type of resources available and providing additional support for non-traditional architectures or computing, thus better serving maintainers and community members.

Amount Requested:

10000

Brief Budget Justification: (Please include hours and/or pay rates)

The budget will be used to pay for development time for key personnel in this grant as follows:

  • Expense: Key Personnel (dev hours)
  • Number of hours: 112 hours (for the project's duration)
  • Total: $ 10,000 USD

Timeline of Deliverables:

We would like to complete this project as soon as possible; however, to implement this with comprehensive community feedback, we are accounting for reasonable response times in our consultations with the conda-forge team. To that end, we would like to propose an estimated implementation delivery at the end of September 2023, with the possibility of finishing earlier if the feedback loop is kept tight and no show-stoppers are found.

Identified deliverables are:

  1. Design an access control strategy with the conda-forge infrastructure team (before July 2023)
  2. Implement the proposed strategy in conda-forge repositories (before September 2023)
  3. . Security review of the Cirun (https://cirun.io/) integration layer to prevent unauthorized cross-access to additional CI resources (before September 2023)

Has someone been identified to carry out the work in the proposal? Please list the name(s) of the person(s) who will be carrying out the work and a short statement (approximately 1 sentence) of why they are qualified.

Yes.

  • Jaime Rodríguez-Guerra <[email protected]>, member of conda-forge core, has sufficient experience in the existing infrastructure and CI workflows powering conda-forge.
  • Amit Kumar, software engineer at Quansight, is the author of the Cirun (https://cirun.io/) integration layer and has set up the GPU backend servicing the experimental prototype CI at conda-forge.

How will someone be identified to carry out the work?

Personnel has already been identified.

Please list the name and email address of a project leader(s) who has approved this proposal.

This proposal is also approved by conda-forge core member Filipe Fernandes <[email protected]>.