I was able to capture this traceback:

correct, I plan to migrate pcc-db1001, before the end of the month

spf records updated

good point @taavi, the inventory endpoint does seem to be supported, but in my testing dotted notation for facts is not supported via PQL.

In T366740#9877740, @Dwisehaupt wrote:

@jhathaway Thanks. I have shifted our codfw hosts to use the new mx-out hosts. That is the secondary datacenter. We'll let it run for a little just to verify and then switch the eqiad hosts over.

@Dwisehaupt let me know if you want to do any canary tests, I'm happy to look at raw emails to assess whether they were successful.

Thanks for opening this @MoritzMuehlenhoff. I expect one of the larger tasks to prepare for puppet 8 will be making our code compatible with the strict settings in puppet 8. We could even explore enforcing those settings in our current puppet 7 installation, once the puppet 5 install base is gone.

In T365395#9865652, @Dwisehaupt wrote:

@jhathaway Question about the routing of mail with these hosts. Currently the civicrm host receives mail from mx1001/mx2001. The inbound mail is dmarc and VERP responses to @donate.wikimedia.org

Will this inbound mail be coming from the new mx-out hosts or is there some other pathway?

Patch added to change wikipedia.org's policy to quarantine.

@Vgutierrez patch works great in prod, thanks!

@ssingh thanks for raising your concerns. I agree that our concerns are similar to those in T231387 and @mark's recommendations are largely applicable.

In T364589#9816477, @Vgutierrez wrote:

acme-chief 0.37 deployed shipping https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/8

Apologies for not providing more context for the log output. I was able to successfully curl the endpoint. However, when requesting the individual files via puppet, I received the above error. After briefly reading the api my understanding is that the file_metadata route needs to be added for individual files. Below is the code that generated the error.

Setting this task to invalid, given that I should be able to retrieve the individual files from the puppet filesystem api endpoint and then create the key files in the order needed by postfix, as suggested by @Volans & @taavi.

Issue (2) should be fixed with the above patch.

Issue (1) was due to leftover timers that should only be run only pcc-db1001.puppet-diffs.eqiad1.wikimedia.cloud. I have deleted the timers to avoid confusion. Ideally we should move our systemd configuration to /etc/systemd/system per the ma page and fully manage that directory in puppet, I'm sure there is already phab task somewhere for this.

ok, I think I understand what is happening now, my apologies for my ignorance on how g10k or r10k uses branches.

Given that this has reoccurred and from the emails you provided looks to be duplication on the application layer I think we need to rope in some people on the Mediawiki side. @larissagaulia do you know who might be a first point of contact?

patch is merged, waiting on a new release, https://github.com/kubernetes/minikube/pull/18233

@bcampbell setting this to declined, please reopen, if this is still a concern

After doing some exploration I don't think emulation is viable due to the complexity and the performance, so blocking this task until T320811 is resolved. In the interim running dcl on WMCS looks to be the best alternative.

@Xover if you could paste the headers of two of the messages that would help, the whole raw text would be the best however. I am curious how identical the emails are, as that would help indicate where they are being duplicated.

part of bookworm upgrade sprint week, but I ran out of time, not currently prioritizing this work.

In T360804#9680160, @akosiaris wrote:

FYI, this is going to be so pretty slow (anecdotal stories speak of standard puppet CI taking up to 15mins instead of a few secs, other workloads vary). See T320811 for the bigger picture, the rest of the tasks in ARM support spell out the various other problems met by developers.

@DBu-WMF the current dmarc monitoring is still a work in progress. ITS has purchased a subscription to dmarcdigests via the security budget, which is currently active, but they are not intending to keep the service long term. However you can open a ticket with ITS to request access, while it is still available. As another route @Jgreen wrote some scripts for ingesting the reports for their own needs and we briefly discussed reusing that tooling outside of fund raising tech, but the work has not been planned as of yet.

In T356920#9672323, @Aklapper wrote:

@jhathaway: Another question is why the task is in S4 (Hardware Procurement) while it seems to have nothing to do with hardware procurement?

Thanks for the poke @RoySmith, ITS obtained this information from Zendesk on how Zendesk's spam marking system operates:

@SLyngshede-WMF let's move discussion to phabricator. I tried the updated patch on my x86_64 hackintosh with minikube 1.32:

patch, https://gitlab.wikimedia.org/jhathaway/dcl/-/merge_requests/2

1. Incident metadata