I’m at Apache Conf in Beijing and a huge theme here is “AI slop PRs ruining open source.” “Every PR you open is spending someone else’s money.” That’s the vibe.
One thing a speaker pointed out is that freeCodeCamp should have an AGENTS.md file not because we’re using agents but because agents that are trying to open PRs will read it.
What do you all think? What should this file say?
One idea is to ask agents to mention “bananas” in their PR, so it will be easy for us to identify PRs that were quickly created with minimal human involvment, since a human would probably remove that before submitting if they were looking.
I’m not saying we don’t want code contributions if they’re surgical and improve our platform without adding a lot of cruft, but the vibe I’m getting from talking with other maintainers here is most of these agent-authored PRs are simply a waste of maintainers’ scarce time and drown out genuine human contributions.
i would actually consider leveraging it to work in our favour:
agent file links to contributing docs
agent file explicitly states to guide the contributor towards making this contribution on their own
astro makes pretty ingestible docs as-is, but if necessary can set up an mcp for agents to consume our contributing documentation
worth bearing in mind however that all of this is trivial to circumvent - which is why ive been using a different solution entirely admitedly i am realising i never communicated this outside of discord…
essentially when a maintainer suspects that a pr is being driven by a human-out-of-the-loop workflow they can slap a Please see Naomi label - and @imsomilg has created an excellent template instructing the contributor to join our discord community and connect with me
this allows me to take on the work of re-onboarding or removing these contributors based on how my discussions with them go - freeing up our community to focus on prs which follow our standards~
Naomi’s idea is actually really good - we even had a success with it today. The contributor headed to Discord, a private room was created for them, and a good solid discussion happened about their AI use, and they left with a clear understanding of the issue.
So now whenever we notice an AI generated PR, we slap the Please see Naomi label and let contributors know we’re pausing further review on it until that happens.
Hi @username
Thanks for your work on this PR! 🙌
Before we can move forward with the review, we'd like you to hop into our [Discord](https://chat.freecodecamp.org/) and have a quick discussion with `@naomi_lgbt` about the approach taken here. There are a few points that would be easier to sort out in a live conversation than back-and-forth on GitHub.
Once you've had that conversation and reached an agreement on the approach, feel free to come back and continue with this PR. 😊
Until then, we'll pause further review on this PR.
The first question you need to ask is do maintainers want to accept AI generated PRs. Some will admittedly be more polished than seasoned contributors. But I think a lot will be low quality or make out of scope changes. Most will be low effort. The quality of the repository will be reflected by the quality of the contributions.
At the moment, maintainers everywhere are beginning to burnout from bottlenecks created by these PRs. One solution is to use AI tools and AI reviews.
I think a way to reframe this is to ask what is the purpose of code contributions. Is it for the contributor to learn about the codebase, read contribution documentation, spend time learning about the repo structure, make conscious contributions?
From what I’ve read freeCodeCamp staff use AI in all sorts of ways to do their work. Plus, making AI PRs gives someone the opportunity to learn and tryout AI. The downside is that the AI does the work, the person doesn’t and they may not really learn anything.
There is an open issue about dealing with AI PRs:
I’m not sure if it is having much effect on contributors.
Agent files are good, but are they really working in the favour of contributors (to learn about open source, engage with contributions, have an actual experience they can remember). Agents can still do the work and a person skims over the output.
There is also the depriortized tag, which effectively filters out low effort contributions.
I suspect contributors are using the PR to check a box on their resume or CV. Which is fine, we all got to start somewhere.
So, is a level of low effort automation acceptable? The tools may become better at contribution, and the person uses the tools.
Maybe consider drafting an internal policy (if it doesn’t already exist) on acceptable AI generated PRs. That way staff and maintainers can make their own comments.
So in conclusion, spend time now dealing with AI and maybe get better quality tools. Put in place more restrictions and maybe gain better contributors. Or a hybrid approach where the contributor becomes adept at using AI.
My guess is that most contributors never see that post.
Even though it is pinned in the issues tab, I think most new contributors to the repo are just excited to get started with open source and jump to the first open issue they see.
If we really want to ensure that folks see it, we could update the PR checklist with an item saying they have read the AI policy here
Then if they don’t check that item off, their PR gets the deprioritized label
the PRs that request the most work from maintainers are those that we want to merge, as it’s those that we review multiple times to catch all the “that could be improved” points
for those that need to be closed because they are a blatant disregard of the contributing guidelines, a maintainer really does not need to do much, with discord open it’s just a slash command from camperchan
it is currently automated, for PRs that are created through the github UI, miss the template checklist, or do not target an issue open for contribution
it’s not the auto-generated PRs that create most work, we just recruited @imsomilg to help with reviewing sprint PRs, which is those that require most time, specially when they are about whole projects (be that adding new projects, or improving whole projects)
