@RoySmith definitely concerning, I asked Bishonen to send me a test email, so I can look through our email logs.

Just had a look and this did indeed go to spam. I have pulled the email out and can allowlist Bish.

@jrbs can you forward me a copy of the original headers, please remove the body of the message, I would like to investigate which part of our mail system tagged it as spam.

In T338032#8899252, @jhathaway wrote:

@jrbs can you forward me a copy of the original headers, please remove the body of the message, I would like to investigate which part of our mail system tagged it as spam.

Done to your Foundation email.

@jrbs there are no spam scores in the headers, which seems strange, did gmail indicate why it was marked as spam?

@jrbs do you have any gmail specific spam settings for the emergency email address?

In T338032#8899554, @jhathaway wrote:

@jrbs there are no spam scores in the headers, which seems strange, did gmail indicate why it was marked as spam?

Not totally sure since I've since approved the email. IIRC, it was something like "It is similar to messages that were identified as spam in the past".

In T338032#8899584, @jhathaway wrote:

@jrbs do you have any gmail specific spam settings for the emergency email address?

I think the best team to ask would be ITS. T&S operates the emergency mailbox but the backend is the domain of ITS.

I think the best team to ask would be ITS. T&S operates the emergency mailbox but the backend is the domain of ITS.

Yes, confirmed. This is accurate, the address is handled in Google:

[mx1001:~] $ sudo exim4 -bt emergency@wikimedia.org
emergency@wikimedia.org
  router = gsuite_account, transport = remote_smtp
  host aspmx.l.google.com [142.251.16.26]

Unfortunately ITS does not use our ticket system :( But you can send a request via https://wikimedia.zendesk.com/hc/en-us/requests/new

Therefore probably nothing can be done here on the Phabricator ticket.

Just to add background that I know Joe knows but others may not, there was a period of time around Dec/Jan where I was aware of a number of emails that were getting caught in the emergency spam filter. I am wondering if ITS might need to take a broader look to help that account - information that might indicate possible spam (i.e. language indicating someone's life is on the line) for the normal WMF staffer is less helpful for emergency@

One other option I see would be to ask for a redirect from emergency@ to emergency@lists, request a mailman list and handle it there. Administrators of the list could then set their own spam filters / regex. But they would have to do so on their own, with the pros and cons that brings.

Thanks @Dzahn, I'll open a ticket with ITS to investigate.

Just to give some additional insight: Emails sent to emergency@ are routed into our Zendesk system and, from there, to our PagerDuty system. This ensures data consistency and (when issues like these aren't happening) that someone from the team is paged to the issue.

If you haven't received a response to an emergency ticket after 30 minutes or so that is a pretty good indicator that something has gone wrong. I'd suggest a good fallback to that is probably emailing our ca@wikimedia.org address or literally pinging me directly on IRC or the Wikipedia Discord, if it's not the middle of the night in SF (I go by "foks" and/or "tzatziki" in both places).

What is SF? When Bishonen is awake, is in fact mostly night in US! Is there special IRC channel for WMF or Emergency? Or use #wikipedia-en-admins?

I am not sure realtime media helps unless we have actual coverage 24/7 with people in multiple timezones handling these?

edit: We are moving away from the actual topic issue though a bit, which was undelivered email.

edit 2: SF is San Francisco, California, USA

In T338032#8902758, @Dzahn wrote:

I am not sure realtime media helps unless we have actual coverage 24/7 with people in multiple timezones handling these?

I hope there is 24/7 coverage. That's certainly what's promised at https://en.wikipedia.org/wiki/Wikipedia:Responding_to_threats_of_harm:

This address is monitored around the clock. Staff will typically acknowledge your email immediately

Hm, this certainly strikes me as a concerning issue (re-surfacing) for the 24/7 service (which it is). It needs to sit in ZenDesk (regulatory reporting requirements are automated in there) but I do wonder whether we could build an automated allowlist that ITS and T&S could rely on - say "email address has been verified to be attached to admin account/an RCler whitelist on a production wiki -> automatically put onto allowlist", or something.

@jhathaway I'm going to respectfully push back on the idea of prioritizing this as "low". Emergency@ is used to report death threats, people considering self-harm, and well, emergencies. Allowing these to get lost in a spam filter with no human oversight seems like a problem which needs to be addressed.

This should likely be escalated to the ITS team, since they handle the Google mailbox this is about.

Since that team doesn't use Phabricator unfortunately, this would mean someone should submit a request via https://wikimedia.zendesk.com/hc/en-us/requests/new

I just opened a zendesk request, briefly describing the problem and linking to this phab ticket. Unfortunately it looks like zendesk doesn't provide any kind of ticket tracking token, so I have no idea how to refer to it.

Thanks. Please share the number (usually five or six digits) of the ZenDesk request here

Sadly, I didn't get any such number. I got a pretty page with a drawing of a bunch of pastel-colored houses and a message thanking me for submitting my request, but nothing that could be used to trace the progress of that request, i.e. no five or six digit number. The request was submitted a few minutes before 2:37 PM when I updated this ticket; I assume it wouldn't be difficult to check the Zendesk logs from about that time to find the ticket in question.

I just submitted another ticket to see what would happen (subject: "this is a test"). Here's a screenshot of the pretty page you get:

@RoySmith The way it's expected to work is that you would get an email that says "Your request (XXXXX) has been received.." where XXXXX would be that number.

@Dzahn I received no such email. Yes, I checked my spam folder.

That's unfortunate :(

Maybe it's set to only respond to @wikimedia.org emails then.

edit: submitted a request to find out if that's true

This topic just came up in another forum and I didn't know what to say regarding status. Could I get an update?

@RoySmith did you ever here anything back from the ticket you opened? Also, you said this was mentioned in another forum, in what context, was there another message marked as spam?

@jhathaway This came up during informal discussions between several Ombuds Commission members earlier today.

No, I never heard back anything from the zendesk ticket. As noted above, I have no way to track that since I never got a zendesk ticket number, and it's not even clear the ticket got opened properly. @Dzahn
maybe you know something?

In T338032#9447464, @Dzahn wrote:

Maybe it's set to only respond to @wikimedia.org emails then.

edit: submitted a request to find out if that's true

I claimed I did this on Jan 9th but now I can't find a mail or ticket number :( sorry about that. Let me try again.

In T338032#8899762, @jhathaway wrote:

Thanks @Dzahn, I'll open a ticket with ITS to investigate.

@jhathaway Did you open one and have the number?

In T338032#9530030, @Dzahn wrote:

@jhathaway Did you open one and have the number?

I couldn't find a record of me doing so either, sorry. I have now opened a ticket:

Your request (#100086) has been received, and will be reviewed by our support staff soon.

I made a new ticket with ITS and I got a confirmation and number:

Your request (#100088) has been received, and will be reviewed by our support staff soon.

hehe, I will tell them to merge mine into yours or something :)

Is there some way I can track those zendesk tickets?

In T338032#9530072, @RoySmith wrote:

Is there some way I can track those zendesk tickets?

I am waiting for a response to the general question whether people outside @wikimedia.org can open ITS tickets. Will let you know soon.

One of you that opened a ticket could try replying to it and copying RoySmith. That might add them to the ticket.

I tried this (CCed and asked to add the email address I got out of LDAP).

This is really frustrating. I've apparently been added to the email thread for the zendesk ticket, but I can't access the ticket itself. The email thread includes URLs such as:

https://wikimediainternal.zendesk.com/hc/en-us/requests/100086
https://wikimediainternal.zendesk.com/hc/en-us/requests/100088

both of which 404 on me:

oops
The page you were looking for doesn't exist
You may have mistyped the address or the page may have moved

I've made myself a zendesk account but when I log in, and look at https://wikimediainternal.zendesk.com/hc/en-us/requests, it says "No requests found" under both the "My requests" and "Requests I'm CC'd on" tabs. Could somebody please get me direct access to the zendesk ticket so I can track the progress?

In T338032#9530131, @Dzahn wrote:

In T338032#9530072, @RoySmith wrote:

Is there some way I can track those zendesk tickets?

I am waiting for a response to the general question whether people outside @wikimedia.org can open ITS tickets. Will let you know soon.

I'm not a WMF ITS rep in any way, but I never managed to create a techsupport@ (aka an ITS ticket) from my volunteer address; it always worked only when I emailed them from my staff address. Otherwise, the messages were seemingly ignored, similar to what happens here.

FWIW, I can't see requests I got CC'ed on (using my staff address) in the zendesk interface; I only see reqs I created myself.

Based on https://meta.wikimedia.org/wiki/User:EBarrios_(WMF), it would appear that Eliza Barrios is in charge of the group that runs zendesk. I have emailed her asking for assistance in getting this sorted out. I'm attaching a copy of that mail here:

Eliza,

Hi. I'm a volunteer working on wikipedia, mostly as an editor, but also as a functionary and occasional software developer. In June of last year, I became aware of an issue where email sent by another editor to emergency@wikimedia.org was never delivered. I opened a phab ticket to see what could be done about that.

Here we are 8 months later and we've progressed to the point that a zendesk ticket has been opened so ITS can work on the problem, but I am unable to access the zendesk ticket so I'm unable to track the progress. Several other people working on the phab ticket have reported similar experiences. We're going in circles. Could I impose on you to take a look at what's going on and get me and any other people who are trying to track this direct access to the zendesk system so we can stay informed of progress?

The tickets in question are:

https://phabricator.wikimedia.org/T338032
https://wikimediainternal.zendesk.com/hc/en-us/requests/100086
https://wikimediainternal.zendesk.com/hc/en-us/requests/100088

Thank you for your assistance.

@RoySmith I also asked ITS if we could use phabricator to communicate, since it is accessible by volunteers.

Is there any progress on this?

Thanks for the poke @RoySmith, ITS obtained this information from Zendesk on how Zendesk's spam marking system operates:

We understand how frustrating it is to have legitimate emails marked as spam. Our spam filtering system relies on a spam-rating service called Cloudmark, whose algorithms identify potential spam. Cloudmark is an industry leader when it comes to spam detection. Unfortunately, we have limited visibility into the criteria driving these ratings, as Cloudmark's processes are external to Zendesk and involve their proprietary algorithms.

If you believe an email was incorrectly marked as spam, we recommend recovering it from the Suspended Tickets view. This helps to train Cloudmark's algorithms to recognize similar emails more accurately in the future. Please note that it may take some time/multiple examples for these algorithms to adjust. Also, this impact may not be permanent as Cloudmark may adjust its algorithms further based on new spam data. To better understand why certain emails are flagged as spam, we recommend reviewing our article: What does "Detected as spam" mean? (https://support.zendesk.com/hc/en-us/articles/4408832769306-What-does-Detected-as-spam-mean-)

If the issue persists after several days of recovering suspended tickets, please notify us and provide several examples including the Message ID and timestamps of the emails that were suspended. The Message ID is shown when opening the message from the Suspended Tickets view.

I don't think attempting to train Zendesk's spam system to correctly recognize legitimate emergency requests is a worthwhile approach. Instead I think we need to consider either removing the spam checking, if possible, or investigate alternatives to an emergency response system which has email systems in its critical path.

@jhathaway thanks for the response. Yes, I agree with you that trying to make zendesk/cloudmark do something it doesn't want to do is not the way to go, so investigating other systems seems like the way forward.

T&S update: Hey all - this is an update on the changes we've implemented regarding the emergency@ inbox, which T&S owns. To ensure optimal support for this critical workflow, we've removed filters and implemented a whitelist to prioritize emails. Additionally, we've incorporated regular checks of the spam folder. We are committed to this process, if anything else comes up please don't hesitate to reach out via ca@wikimedia.org or ping me or @jrbs. Thanks.

Yesterday afternoon, I sent email to ca@wikimedia.org to make sure they were aware of this post on enwiki. After not getting an acknowledgment a few hours later, I followed up with another email to ca@, copied to emergency@, asking for confirmation that they received it. I got no response. That was about 12 hours ago.

So what's going on here? Is this another instance of mail to these mailboxes falling into spam folders?

@Nahid would you be able to take a look into @RoySmith most recent message?

Hey, let me investigate and get back.