---> T382610

edit: deploy-1006.devtools also affected

Sounds great! Keep in mind that it might not boot because the device names change. If that happens we have to fix that via console. (re: the warning section under https://wikitech.wikimedia.org/wiki/Ganeti#Adding_a_disk)

Thank you very much @Marostegui . That was above and beyond. You didn't have to check the service for us. It's really appreciated that you did anyways:)

As an organization, we strive to use open source tools over proprietary ones, although we use proprietary or closed tools (such as software, operating systems, etc.) where there is currently no open-source tool that will effectively meet our needs.

We have had the temporary DNS issues in other projects (not integration) as well in the past. And it doesn't seem to match the super low number (five nines) that Cathal saw in his tests. I am wondering if there is something a certain subset of projects has in common that other projects don't have.

I think the options are pretty much:

Since the data is generated and uploaded to doc hosts automatically it means at all times doc1003 and doc2002 should be pretty identical when it comes to disk space.

I remembered we had monitoring somewhere that would check conntrack table size.

I am out now until Jan 2. There is no rush to this regardless of the choice. You can either just replace the IP and resolve this ticket or you can leave it open. Up to you. Cheers and thank you!

In T377643#10393927, @Marostegui wrote:

I think we are mixing two different things here.

We have been evaluating software for a refreshed codesearch and https://www.sourcebot.dev/ seems like a viable candidate.

The usual effect after a GitLab security upgrade (T381969)

Any thoughts yet on where you actually want to host this?

Thanks @Clement_Goubert :))

https://secure.phabricator.com/book/phabcontrib/article/version/ says "In general, we use git commit hashes as version identifiers, so you can identify the version of something by running git show and copy/pasting the hash from the output."

Thanks for doing this @Pppery

See class profile::microsites::monitoring in the puppet repo for examples to copy/paste/adjust to get the Blackbox check.

Both T350793#10378435 and today T350793#10394688 happened and caused realtime debugging. This seems to be at odds with the claim that nobody uses it anyways.

example config to pull gitlab repos, limited to repos under repos/sre.

regarding the actual size: The data points I have right now are:

justification: demo/testing/staging for T268199

Thinking about it, this could also be granted in the project called "codesearch". It would basically make no difference for us in which project we have it.

The WCQS issue has been solved by this revert to firewall config on the legacy miscweb VMs:

example config to pull gerrit repos from Wikimedia. Note the wildcard patterns appear to match only exactly one level or 2 levels of repo names, not all.

These refer to the source repo at https://we.phorge.it/source/phorge/browse/master/src/applications/harbormaster/

Thanks for reporting.

Just dumping some data here for a comparison. The current codesearch VM, codesearch9 is a g4.cores4.ram8.disk20 running Debian bookworm.

Phabricator (Phorge!):) prod host rebooted to unload ferm modules. Phorge hosts done now.

One thing to answer here would be how you would know who actually is WMDE staff. There used to be a public page that lists them but then that stopped existing.

We just installed package version 3.3.8-2~deb12u2 and this should be fixed now. Please let us know how it looks.

Thanks @Pppery !

I mean the phab host. We want to be able to switch which of them is the active server if necessary while also being sure we never ever write to the DB from 2 hosts at once.

I realize the wiki creation tickets and their subtasks are created automatically by software. In the past I also had ideas how to slightly change what is a subtask of what etc for certain things.

In T381464#10386635, @HShaikh wrote:

For some clarity. The request is for Chris to be able to eventually run jupyter notebooks.
So he is requesting access to the analytics-privatedata-users group in the analytics groups with SSH and kerberos access.

☑ https://app.betterworks.com/app/#/profile/710331

Cool! I tested the downgrade and upgrade with APT as well on lists2001. Worked both ways.

Since we could not test if the service starts on list2001 (fails because it can't talk to the DB), I created a test instance in the cloud VPS project called mailman. Manually copied the deb over, installed it and let apt fetch all the dependencies. Then installed postfix with "localonly" config and I could succesfully start the mailman3 service.

I installed the package manually on the passive host, lists2001.

We could but if we do that and just close the ticket it means we could never fail over to this host as the active phab host. And we still want to solve that.

Despite what I expected or said previously, it turns out the dumps are currently created on the active server, so not this one.

In T377045#10371355, @MoritzMuehlenhoff wrote:

I have backported https://gitlab.com/mailman/mailman/-/commit/353a2adf55381c09bf7c2bc7ddb532a06e3934f8 and built as mailman3_3.3.8-2~deb12u2+wmf1.

Yes, an RO user is what we would like to have. The part that we are using m3-slave is basically just because that was adviced as the easiest way to get the same thing as an RO user.

Since we confirmed that Phabricator dumps are not actually broken, they are generated on the active server, I can lower priority on this.

after merging https://gerrit.wikimedia.org/r/1092841 a puppet dependency cycle happened (only on the 2 phab hosts, not on the other 178 scap::target hosts).