Camilla Moraes, Product Manager at GitHub, initiated a discussion regarding the incorporation of a feature on GitHub to automatically block junk pull requests created by AI assistants. These pull requests are submitted without manual review and fail to meet quality standards. Such unvetted contributions put an extra strain on maintainers, who are compelled to sift through irrelevant code.
Immediate solutions to address this issue include the option to swiftly delete pull requests through the web interface (deleting them without them appearing in the history, as opposed to marking them as closed) and utilizing custom permissions for submitting pull requests. This would allow repository owners to restrict changes only to contributors who have previously made alterations.
In terms of long-term remedies, suggestions include enhancing the permission model and equipping maintainers with the means to establish flexible guidelines dictating who can create and review pull requests, as well as specifying the criteria pull requests must meet. Additionally, there is a proposal to leverage AI technology to assess the adherence of submitted changes to project rules and quality standards (as outlined in the CONTRIBUTING.md file) and to identify and distinguish changes made with the assistance of AI.
Among the proposals raised during the discussion is the implementation of a filter that prohibits the submission of pull requests without first initiating discussions explaining the rationale behind the proposed changes and notifying maintainers of pull requests from new contributors only after these requests have successfully cleared tests in the continuous integration system.
According to statistics from a key developer of the genkit framework, only one out of ten AI-generated changes meets the criteria for opening a pull request. A participant in the Azure Core Upstream project summarized the main concerns of maintainers:
- Violation of the trust model during reviews, as reviewers cannot ascertain if the code was authored by the submitter or understand its essence.
- Pull requests from AI assistants may appear structurally sound but could