Page MenuHomePhabricator
Create Task
Maniphest T262499

SpecialMuteSubmit: add schemas to EventLogging whitelist
Closed, ResolvedPublic
Actions

Description

Proposal
EventLogging schema for SpecialMuteSumbit is only kept for 90 days. We hope we can sanitize user level info (ueragent, ip, geocoded_data) and keep action event logs (defined in https://meta.wikimedia.org/wiki/Schema:SpecialMuteSubmit) and pipeline to event_sanitized database.

ipuseragentuuidseqiddtwikiwebhostschemarevisiontopicrecvfromeventgeocoded_datayearmonthdayhour
sanitizedsanitizedkeepkeepkeepkeepkeepkeepkeepsanitizedkeepkeepsanitizedkeepkeepkeepkeep

The description of the kept columns:
uuid : Unique event identifier
seqid: Udp2log sequence ID
dt: datetime
wiki: wiki project
webhost: Request host.
schema: Title of event schema
revision: Revision ID of event schema
recvfrom: Hostname of server emitting the log line
event: The encapsulated event object

schema doc:
https://meta.wikimedia.org/wiki/Schema:EventCapsule
https://meta.wikimedia.org/wiki/Schema:SpecialMuteSubmit

Stages:

  • Legal approval for data retention
  • Add schemas to EventLogging whitelist

Details

SubjectRepoBranchLines +/-
Add SpecialMuteSubmit schema to EventLogging whitelistanalytics/refinerymaster+9 -0
Customize query in gerrit

Related Objects

Event Timeline

jwang created this task.Sep 10 2020, 1:06 AM
jwang updated the task description. (Show Details)
jwang changed the visibility from "Public (No Login Required)" to "All Users".
Niharika changed the visibility from "All Users" to "Public (No Login Required)".Sep 10 2020, 7:36 PM
Niharika moved this task from Untriaged to Analytics on the Anti-Harassment board.
jwang updated the task description. (Show Details)Sep 14 2020, 11:45 PM
LGoto assigned this task to jwang.Sep 15 2020, 5:08 PM
LGoto triaged this task as Medium priority.
LGoto edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
LGoto moved this task from Next 2 weeks to Blocked on the Product-Analytics (Kanban) board.
APalmer_WMF added a comment.Sep 15 2020, 6:27 PM

Approved by Legal.

jwang updated the task description. (Show Details)Sep 15 2020, 6:41 PM
jwang updated the task description. (Show Details)
jwang added a comment.Sep 15 2020, 6:49 PM

@APalmer_WMF , Thanks.

To make it more clear, I added the description of the columns in the proposal.

jwang updated the task description. (Show Details)Sep 15 2020, 6:49 PM
gerritbot added a comment.Sep 17 2020, 11:46 PM

Change 628235 had a related patch set uploaded (by DannyS712; owner: Jenniferwang):
[analytics/refinery@master] Add SpecialMuteSubmit schema to EventLogging whitelist

https://gerrit.wikimedia.org/r/628235

gerritbot added a project: Patch-For-Review.Sep 17 2020, 11:46 PM
DannyS712 subscribed.Sep 17 2020, 11:46 PM
jwang moved this task from Blocked to Doing on the Product-Analytics (Kanban) board.Sep 21 2020, 4:02 PM
jwang added a subscriber: • JFishback_WMF.Sep 22 2020, 10:13 PM

@JFishback_WMF, please let me know if you have any concern.

gerritbot added a comment.Sep 25 2020, 7:48 PM

Change 628235 merged by Nuria:
[analytics/refinery@master] Add SpecialMuteSubmit schema to EventLogging whitelist

https://gerrit.wikimedia.org/r/628235

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2020, 8:10 PM
• JFishback_WMF added a comment.Sep 29 2020, 4:03 PM

Hello @jwang same comment as here. Is it necessary to keep the dt field? What is the purpose of keeping this data long-term (please feel free to reply off-task if it is sensitive)? Thanks!

• JFishback_WMF added a project: Privacy Engineering.Sep 29 2020, 4:03 PM
• JFishback_WMF moved this task from Incoming to Waiting on the Privacy Engineering board.
jwang added a comment.Sep 29 2020, 5:06 PM

@JFishback_WMF , thanks for the review and comment. Please find my answer here since they are the same topic.

jwang mentioned this in T261461: Capture special mute events in Prefupdate table [4 hour spike].Oct 15 2020, 4:36 PM
jwang moved this task from Doing to Blocked on the Product-Analytics (Kanban) board.Oct 27 2020, 5:13 PM
jwang moved this task from Blocked to Doing on the Product-Analytics (Kanban) board.
kzimmerman moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.Nov 30 2020, 10:24 PM
kzimmerman subscribed.

Pending final review from @JFishback_WMF

• JFishback_WMF added a comment.Dec 14 2020, 10:22 PM

Generally, the best practice is to minimize the data that is collected - especially high resolution data. That said, this data appears to be LOW risk, and per @jwang there is a countervailing interest in collecting the high resolution data in order to retain sequencing and other analytical uses.

• JFishback_WMF moved this task from Waiting to Completed on the Privacy Engineering board.Dec 14 2020, 10:22 PM
jwang closed this task as Resolved.Dec 16 2020, 12:26 AM
jwang updated the task description. (Show Details)

Thanks for the review and suggestion. Mark it as resolved.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy