Testing a New Stalebot in GitHub's Community Discussions #70478

queenofcorgis · 2023-10-18T16:12:30Z

queenofcorgis
Oct 18, 2023
Maintainer

TL;DR - What is happening?

We are working to improve the quality of content across the GitHub Community Discussions. For the next 30 days we are going to perform a test using a custom action to label Question discussions that have not had any activity for 60 days as stale. The action will apply the label stale and comment to inform the original poster of the stale discussion. Sample draft copy is included at the bottom of this announcement.

📢 Feedback is requested on this initial test. Please comment on this post and let us know what you think!

What Discussions will be impacted?

The action will label discussions that have been labeled Questions and have not had and activity in the past 60 days as stale.
The action will post the sample copy (included below) as a comment on the discussion to alert the original poster, and request they take action to resolve the question. That automated reply should prompt users to return to add more information, select a reply as the solution, or close the question as out of date. I'll add some example copy we can use below.
This will only be applied to discussions in the Copilot, Projects and Issues, and Accessibility categories as a part of this initial test.
This action will not close the discussions automatically.

What is a Stalebot?

You can use GitHub Actions to comment on or close issues that have been inactive for a certain period of time.

Here is more info on this from the GitHub Docs: Closing inactive issues.

We're working with GitHub engineers to test this action in discussions!

Why are only limited categories impacted?

We have a hypothesis that using the action to close the loop on open and stale questions in the Community Discussions will improve the quality of content overall, but we want to test assumption before we risk disrupting the entire community discussions experience.
The initial test includes categories with high engagement, dedicated community managers, and a high level of involvement from GitHub engineers.
We are starting with an initial test with Discussions that are 60 days stale. We want to see how the age of posts impacts overall engagement and resolution rates. We track how long a discussion is open without a response and without an answer so we'll be able to track data on this test and use that to inform our next steps.
We want to hear from you! This is an experiment to benefit our users and the quality of content. If it doesn't work, we can roll it back. If it does, we can move forward and apply this across all categories.

What will happen after the trial period?

The team will have an ongoing evaluation of how the stalebot impacts the response and resolution rates on exiting discussions.
If we see positive indicators based on data that will be one indicator that this worked
We will evaluate the feedback on this discussion. If you love it or hate it, we want to know! Please comment with ways you think we can improve the experience.
If we deploy this broadly, we'll provide users with a hot-to guide here in the Community Discussions so you can apply a similar action to your own repo discussions 🚀

Example Stalebot Copy:

🕒 **Stale Discussion Alert** 🕒

This Discussion has been labeled as stale by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as "out of date" at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This stale notification will only apply to Discussions with the `Question` label. To learn more, see our recent stalebot announcement.

Thank you for helping bring this Discussion to a resolution! 💬

ferdnyc · 2023-10-20T11:33:07Z

ferdnyc
Oct 20, 2023

@queenofcorgis Since you're showing the sample copy, presumably that's meant to invite input, so I have a suggestion:

Note: This stale notification will only apply to Discussions with the Question label. To learn more, see our recent stalebot announcement.

...The message could just link to the "recent stalebot announcement" (that'd be this discussion, #70478), instead of sending us on a scavenger hunt.

2 replies

GuiSousa2910 Jun 16, 2024

yes

timmc-edx Aug 9, 2024

Looks like they've updated it to do so now.

ant-techs · 2023-10-26T10:07:23Z

ant-techs
Oct 26, 2023

Hi How can I talk to the creator/Dev of Zello as I have a proposition for him, if you could get in touch at [redacted] I would be grateful to talk with you, Best regards, Anton.

1 reply

queenofcorgis Oct 26, 2023
Maintainer Author

Hi @ant-techs ,

I've removed your private information from your comment as posting identifiable information is not allowed, per our Code of Conduct.

If you're asking how to use the piece of software that you found in a repository on GitHub, the best way to contact the maintainers of that software is to:

Check the README for instructions on how to operate the software and pointers to documentation or troubleshooting info.
Check the SUPPORT file, if one exists, for instructions on how to best contact the maintainers for support.
Check the CONTRIBUTING guide, if one exists. Sometimes if there isn't a SUPPORT file, the CONTRIBUTING guide will give instructions on how to contact the maintainers for support.

All of these documents can be found, if they exist, in the repository where you found the software itself.

Marcono1234 · 2023-10-31T22:33:40Z

Marcono1234
Oct 31, 2023

Thanks for being so considerate by only limiting it to the "Question" category and not closing discussions directly.

The stale action (and other similar actions) are known to be quite disruptive in some cases, see actions/stale#719, when they close issues where the maintainer has simply not responded yet. Or in this case here, acting on questions (possibly even well written ones) where no one ever commented on.

As you mentioned, many of the questions here are answered by GitHub staff or at least someone from the community. But some also seem to have no answer at all (discussions with label stale and only a single comment). In that case only action 1, "Close as Out of Date", would be relevant.

Do you think that is a problem, and will you try to address that?

0 replies

ant-techs · 2023-11-02T12:59:43Z

ant-techs
Nov 2, 2023

1 reply

ferdnyc Nov 2, 2023

@ant-techs

Then you should probably change your GitHub password, because 7 days ago your account posted a message to this discussion that started with, "Hi How can I talk to the creator/Dev of Zello as I have a proposition for him..."

jsoref · 2024-01-23T20:51:14Z

jsoref
Jan 23, 2024

@queenofcorgis sadly, this is in the announcements category, because I'd really like to see the action mark this discussion as stale.

I'd like to hear if you in fact ran this action and what the results were.

I personally hate the stale bot. (That I send PRs to it is not an endorsement, it's awful software for a lousy purpose.)

There appear to be two constructive pieces of feedback here:

If you're running an experiment, you should link to the announcement about the experiment: https://github.com/orgs/community/discussions/70478#discussioncomment-7338056
A critique (similar to mine) of the feature: https://github.com/orgs/community/discussions/70478#discussioncomment-7440236

There's also a PR relating to this discussion:
#69778

One problem with bots is that they make a mess of sorting (by last modified) by bubbling up stale items. Sure, it might breathe be life into a thing, but, for a system built around searching, that isn't worth doing. If someone's searching for a thing, they'll find or not find it based on their search and the text of the item.

A better approach would be to improve the UX flow so that authors are encouraged to yes/no answers as they arrive.

5 replies

jricciardi May 10, 2024
Maintainer

Hi @jsoref! I can weigh in a little bit here now that we have made some updates and have more data:

With the exception of a few edge case frustrations, we are seeing a lot of overall user acceptance for the stalebot, especially after reducing the "stale" language in favor of "inactive" and "dormant" language.

Additionally, and the numbers change daily, about 5-8% of Discussions are brought to resolution after prompting from the bot; about 1-2% are discovered to actually be product feedback that was incorrectly labeled as a question (this is a clear source of confusion and frustration for some users and something we're paying close attention to); and otherwise most Discussions never get new responses or activity.

A better approach would be to improve the UX flow so that authors are encouraged to yes/no answers as they arrive.

This is great feedback! If we find evidence this type of flow would improve outcomes for the Community, we may escalate this idea.

jsoref May 10, 2024

I've tripped on the question category thing. The bot didn't explain how to recategorize my feedback.

jricciardi May 15, 2024
Maintainer

Currently this is something we primarily rely on Community Managers to action on, and don't give wide permissions to avoid malicious users juggling useful Discussions around.

jsoref May 15, 2024

Well. I have a bunch of items where the bot is complaining. They were generally meant as product feedback, not "questions".

I just walked home from work because I'm still recovering from a concussion, so if a community manager could please fix mine, I'd appreciate it. I can't find the one that was just touched today, and my headache is getting worse.

ferdnyc May 18, 2024

@jsoref

One problem with bots is that they make a mess of sorting (by last modified) by bubbling up stale items.

That's true, though the stalebot does also label the discussions it wades into, helpfully. That means there's always the possibility of adding -label:inactive to the search/filter terms, in order to exclude those items.

It's not ideal, but it can at least help mitigate some of the undesirable (or at least, debatably-desirable) effects.

timmc-edx · 2024-05-10T16:13:50Z

timmc-edx
May 10, 2024

A bit of feedback: I'm here because the stalebot just replied to a question I've had open for some months about the GitHub API. It suggests three courses of action, all of which are frustrating to read:

"Close as Out of Date": Nope, I still have the problem.
"Provide More Information": Already gave all the info I have.
"Mark a Reply as Answer": No one has answered, so there's no reply to mark.

The only useful action I could take would be something like "Do $X to get a GitHub employee to look at your question", but that's not one of the suggestions. :-) So the bot is just reminding me that there's a gap in GH's API that I can't do anything about, and there's no one to contact, and no one has responded... which doesn't feel great.

Anyway, I don't have a specific thing I think you should do differently (with regards to the bot at least) but I did want to share that the net effect was very annoying.

13 replies

jsoref May 16, 2024

I see a Product Feedback tag which seems reasonable.

timmc-edx May 16, 2024

Hmm! It wasn't there when I asked. Maybe it got manually labeled? Anyway, I guess the process works somehow. :-)

jsoref May 16, 2024

https://github.com/community/community/blob/main/.github/workflows/label-templated-discussions.yml
https://github.com/community/community/actions/runs/9101979677/job/25020504711#step:2:1

jricciardi May 17, 2024
Maintainer

@timmc-edx Yep, that's applied by the Action. We love a good Action here :)

And thank you for submitting that as product feedback! 🚀

jsoref May 30, 2024

This is going to happen to me a lot. I don't think that "Question" vs "Product Feedback" made such a distinction to me when I filed things, so I bet a fairly large number of my items will get this noise and result in me refiling things.

I viewed "Question" as "why is this product doing X?" or "could this product do Y better?". I imagine I'd use "Product Feedback" for "Z is broken".

I have a headache and an not going to try to do a systematic review of all of my discussion posts. -- it's possible that the method I used to create them mattered.

Today I'm using a mobile browser (where the discussion category was not as visible and easy enough to skip past), whereas for filing issues about GitHub Mobile, I tend to use GitHub Mobile, and otherwise I'll generally be using a desktop browser.

https://github.com/orgs/community/discussions/81741

jsoref · 2024-05-15T17:01:46Z

jsoref May 15, 2024

https://github.com/orgs/community/discussions?discussions_q=is%3Aopen+label%3AQuestion+%22Discussion+Activity+Reminder%22

ferdnyc · 2024-05-29T22:44:31Z

ferdnyc May 29, 2024

As I commented there, your second link actually turned out to be an argument for the stalebot. Its poking of that discussion ultimately led to additional comments being posted and, from those, an answer being selected.

jsoref · 2024-05-30T21:21:41Z

jsoref May 30, 2024

@ferdnyc: I agree with @MasterInQuestion, this wasted a considerable amount of my time as well as the questioners and anyone subscribed.

I'm only adding these admittedly fairly useless comments because the bot is asking me to do so. -- In the case of the selected answer, it wasn't technically useless as it pointed to how the Questioner themselves solved the problem (and when I suggested marking something as the solution, I really meant the Questioner's post saying they'd solved it, more than my own response) -- It's true that the other post was more valuable to the next person as it provided an explanation as to why that change was correct, but...

Worse, each of my interactions risks someone accusing me of misbehaving. Whether that's link spam, abuse, gaming, ...

I don't need/want to take any of these actions. But, I'm invested in getting my tickets addressed. -- And they won't be addressed if they're closed by a bot.

It's true that some tickets may make slight progress. But there are/have-to-be better approaches to achieve that aim.

Contributors who positively improve tickets (by either answering questions, or marking questions as answered, for which answers aren't later rejected by the original poster) could be given action-credits or something else of value within the GitHub ecosystem (I certainly could use those credits). Obviously if one went this way, abuse would have to result in people not being able to receive such rewards, but that's also fairly easy to administer.

jricciardi · 2024-05-30T16:48:00Z

jricciardi May 30, 2024
Maintainer

Hey there @adamlui - It looks like this Discussion was originally posted as a Question and then you edited to be a Bug, but it still had the Question label on it, triggering the bot. That's an edge case we'll have to think through solutions for. Thank you for flagging!

I've gone ahead and relabeled that Discussion for you and removed the inactive label 👍

jsoref · 2024-05-30T18:12:10Z

jsoref
May 30, 2024

@queenofcorgis / @jricciardi: I've refiled a couple as a courtesy to your team (https://github.com/orgs/community/discussions/126580, https://github.com/orgs/community/discussions/126631, https://github.com/orgs/community/discussions/126632).

I'd appreciate it if you were to edit the labels for these remaining three from Question to Product Feedback:

https://github.com/orgs/community/discussions/60448
https://github.com/orgs/community/discussions/58143
https://github.com/orgs/community/discussions/39447

Sincerely,
Josh Soref (@jsoref 🤕 not recovering well from a concussion, and not appreciating unpaid busy work)

0 replies

flodolo · 2024-06-08T04:44:50Z

flodolo
Jun 8, 2024

Here's another example: https://github.com/orgs/community/discussions/47855

It's a question — potentially a bug for the documentation once there's an answer — but none of the options offered is suitable.

Also, not particularly useful that one of suggestions is to Close as Out of Date when the bot already closed the discussion (and I can't do any change to labels).

0 replies

timmc-edx · 2024-06-16T00:05:37Z

timmc-edx
Jun 16, 2024

The announcement reads, "This action will not close the discussions automatically." But I just had a question closed by the bot: "Closed #103026 as outdated." Was that action actually performed by a human, or is the bot now being used to close older issues?

6 replies

jsoref Jun 16, 2024

@queenofcorgis, @mgriffin, @jricciardi could one of you please update this announcement?

-This action will not close the discussions automatically.
+A new action added on April 12, 2024 will close the discussion automatically if no more comments are added after the warning.

timmc-edx Jun 17, 2024

Ugh, thanks for the reference. Stalebots like this are legitimately one of the worst things about "maintainership" as practiced in many GitHub repositories.

timmc-edx Jun 17, 2024

To expand on this: Any community that closes out questions as "outdated" or "old" is a community where I am less likely to bother asking questions or posting issues in the first place, because my work in making well-formed, readable, reproducible posts is just going to be thrown away in a month or two. It's extremely discouraging.

When I see a stalebot, I assume the maintainers are unwilling to say "no" to feature requests and that there's no one around capable of answering questions.

Marcono1234 Jun 18, 2024

@jsoref, it is probably more than that which is outdated:

At this point this doesn't seem to be a "30 days test" anymore, but possibly something permanent.
It seems to apply to more than the mentioned discussion categories (see source).
The label is not called stale anymore but instead inactive.

Maybe it would be best if they completely updated the description of this announcement.

This comment was marked as off-topic.

Sign in to view

ferdnyc · 2024-09-05T20:53:12Z

ferdnyc Sep 5, 2024

I only looked at the second one, and it was a good close. The GitHub community discussion forums are not your blog, and they are not required to host your rambling musings on the divine (which 0 other people had engaged with, in the entire time the discussion was open) indefinitely. There was no "discussion" happening.

Seriously, sign up for a Blogspot account and you can create a bog titled "cross-repo symlinks"¹ and post as much as you like about anything you like. Just remember to leave comments open on each post in perpetuity.

Notes

#NoSuchThing. (You're right that git submodules are the closest analogue, but wrong that they can or should be used with such granularity. With submodules you link to a specific commit of an entire repo, that's the only way it makes sense.)

novium2 · 2024-09-06T23:35:00Z

novium2
Sep 6, 2024

Retiring this post as it is over 6 months old. If you have any questions, consider starting a new post.

0 replies

This comment was marked as off-topic.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment has been minimized.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Testing a New Stalebot in GitHub's Community Discussions #70478

Uh oh!

queenofcorgis Oct 18, 2023 Maintainer

TL;DR - What is happening?

What Discussions will be impacted?

What is a Stalebot?

Why are only limited categories impacted?

What will happen after the trial period?

Example Stalebot Copy:

Replies: 30 comments · 41 replies

This comment was marked as off-topic.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as off-topic.

Uh oh!

Uh oh!

Uh oh!

queenofcorgis Oct 26, 2023 Maintainer Author

This comment was marked as spam.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as off-topic.

This comment was marked as off-topic.

Uh oh!

Uh oh!

Uh oh!

jricciardi May 10, 2024 Maintainer

Uh oh!

Uh oh!

jricciardi May 15, 2024 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jricciardi May 17, 2024 Maintainer

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

This comment was marked as off-topic.

Uh oh!

jricciardi May 30, 2024 Maintainer

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as spam.

This comment was marked as spam.

Uh oh!

queenofcorgis
Oct 18, 2023
Maintainer

Replies: 30 comments 41 replies

queenofcorgis Oct 26, 2023
Maintainer Author

jricciardi May 10, 2024
Maintainer

jricciardi May 15, 2024
Maintainer

jricciardi May 17, 2024
Maintainer

jricciardi May 30, 2024
Maintainer