This is effectively done, what's left to do on my end is put sandbox/filippo/pontoon-puppetserver branch for review and get it merged, which I'll do in July when I'm back from vacation

503s are gone as of ~12:20 UTC

In T367064#9884801, @Peter wrote:

Thank you @fgiunchedi . If I do the work tomorrow Thursday, do you have time to merge it during the day? I can stop the collections early in the morning CET and then you can do your thing when you have time and sync with me on Slack/IRC?

hey @Peter, sure that seems reasonable to me. I have prepped https://gerrit.wikimedia.org/r/c/operations/puppet/+/1042223 to be merged whenever the server is ready to be switched over on your end! I'll be out starting next week until the end of June; though other o11y folks can merge the patch if things are good to go before I'm back. HTH!

{{done}}; resolving though reopen if sth is amiss

Indeed using envoy_cluster_upstream_rq is not going to fly; that is a metric with a lot of cardinality and can't be reasonably used as-is. Make sure you are filtering on more tags to get a smaller result back, depending on needs. For comparison a metric looks like this:

We've definitely made progress here with various limits in place, resolving

In T360895#9875773, @Jhancock.wm wrote:

I got the ram out of the servers. I can do the addition tomorrow morning at 8am CDT (1600 UTC) or another time that works for you.

Yes tl;dr is unexpected high database lag during T367019: Switchover s2 master (db2204 -> db2207) cc @ABran-WMF

I'm optimistically resolving the task, though please feel free to reopen if sth is amiss and thank you for your report

Calling this one done, debian package is uploaded and container updated

@Benwing2 things are back to normal!

Thank you for the report @Benwing2; we're experiencing an issue with some wikis including enwiktionary : https://www.wikimediastatus.net/incidents/94wcdd42gzxz

I've silenced BenthosKafkaConsumerLag alert for now re: this until we investigate and fix

In T360895#9871175, @Jhancock.wm wrote:

I don't have any new on hands. But I can pull some of the extra dimm from the decommissioned servers. I'll leave enough to make sure the servers being recycled are still operational if that's what we want to do. Sound good?

I didn't realize it at the time, though codfw in T354685 got 192GB per host, whereas a week later in T354684 eqiad got 384GB per host.

I tested the package above in pontoon and it works as expected, I'll be importing it into apt and rolling it out to production/baremetal early next week

Thank you @MatthewVernon; I'll be trimming the retention further

I have refreshed the Debian packaging and pushed a new packaging-wikimedia branch to the gerrit repo for prometheus-statsd-exporter; the resulting package is available at /var/cache/pbuilder/result/bookworm-amd64/prometheus-statsd-exporter_0.26.1-1_amd64.deb on build2001. Note that I've patched the source to also accept the --statsd.relay-address argument (as opposed to upstream's --statsd.relay.address) to ease upgrades. Once we have the new version rolled out we can change puppet to use upstream's flag

In T365265#9860976, @colewhite wrote:

In T365265#9858015, @Joe wrote:

1. As for one release per MediaWiki version, it seems to me like it could be pretty wasteful: almost 90% of the metrics traffic is from group2, for instance. We'll need to run some maths on the amount of resources we'd need, but I think we should explore alternatives, specifically:
   1. Look at contributing to upstream statsd-exporter to add a switch allowing to not discard metrics with new signatures, given prometheus won't complain about it AIUI

The need for a per-group statsd-exporter is necessary only if we continue to use statsd-exporter v0.9.0. Upstream enabled the ability to have inconsistent label sets in v0.10.2 I suggest we upgrade statsd-exporter and eliminate the need for per-group exporters: T302373: Upgrade prometheus-statsd-exporter

In T365265#9858015, @Joe wrote:

A few thoughts on this:

1. I think using daemonsets is a better option than a deployment. It will ensure UDP fire-and-forget communication (which is in itself scary in k8s with all the layers of indirection) happens locally to the physical node. For the pods to connect to it, we'll just have to pass down the node ip as an env variable to php-fpm, using the downward api to pass status.hostIP.
2. This has the added advantage of reproducing the load pattern we have on bare metal, of course
3. As for one release per MediaWiki version, it seems to me like it could be pretty wasteful: almost 90% of the metrics traffic is from group2, for instance. We'll need to run some maths on the amount of resources we'd need, but I think we should explore alternatives, specifically:
   1. Look at contributing to upstream statsd-exporter to add a switch allowing to not discard metrics with new signatures, given prometheus won't complain about it AIUI

I'm not sure exactly what happened, though while working today on {T366555} centrallog1002 md1 raid wouldn't come up cleanly. I've assembled it with three disks and then put back the fourth; also correcting this mismatch in the process

Also might be related: https://github.com/rsyslog/rsyslog/issues/5215 https://github.com/rsyslog/rsyslog/issues/5176

Upstream issue: https://github.com/rsyslog/rsyslog/issues/4186

Change is deployed, not a permanent fix though at least the ongoing toil is reduced now

In T365111#9820947, @WMDE-leszek wrote:

hello, a silly question from an uninitiated: will the historic "wrongly labelled" metric data will be re-bucketed, or are we to assume that there will be no metric data for the period in time in question?

I can't authoritatively answer the questions though IIRC from my chat with @Clement_Goubert on using mesh/envoy vs not it was for symmetry with the rest of mw. To be clear: I don't feel strongly either way, whichever is best practice in this case works for me (ditto for the first question FWIW)

In T365111#9820828, @daniel wrote:

The fix has been backported and deployed. It should now be safe to re-enable the metrics.

We had a peak of ~700GB used, so we're fine. Wrapping up T361229 and T359449 will eventually lead us to a raid0 of ~2TB

In T343529#9776787, @gerritbot wrote:

Change #1025682 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: use longer-expiration pki client certs for k8s

https://gerrit.wikimedia.org/r/1025682

We current have tracing enabled for cxserver and citoid in staging. As a first step and to gain confidence I'll enable tracing for those in production.

For the folks subscribed to this task and interested in beta-testing, please see sandbox/filippo/pontoon-puppetserver branch and its README.md: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/sandbox/filippo/pontoon-puppetserver/modules/pontoon/

I've temporarily blackholed the metrics from graphite, though of course mw should stop sending them. cc @hashar @daniel @BPirkle

Splitting up curator work SGTM!

This is done! Latest chart version is deployed

In T364645#9790603, @herron wrote:

In T364645#9789322, @fgiunchedi wrote:

A bit of a different route, though I can't remember if we have tried telling pyrra filesystem about "prometheus" being thanos-rule on localhost? i.e. --prometheus-url http://localhost:17902/rule/ ? In other words let pyrra filesystem effectively do the reload

This would be ideal, although as-is the prometheus-url is used to build links for each panel in the pyrra UI to prometheus (thanos in our case) for detailed query/metric view.

FWIW I opened https://github.com/pyrra-dev/pyrra/issues/986 about this last year which got some support but hasn't made any progress yet.

An example task of such migration is https://phabricator.wikimedia.org/T246998, which basically translates to:

• provision a new oidc client for prometheus in idp
• introduce a prometheus apache configuration to proxy requests for prometheus-SITE.wikimedia.org to oauth2-proxy
• configure oauth2-proxy to proxy authenticated requests to prometheus.svc.SITE.wmnet

A bit of a different route, though I can't remember if we have tried telling pyrra filesystem about "prometheus" being thanos-rule on localhost? i.e. --prometheus-url http://localhost:17902/rule/ ? In other words let pyrra filesystem effectively do the reload

From my POV prometheus in magru is live and working, see also https://prometheus-magru.wikimedia.org/

I've run into this today, what seems to happen is the following:

In T350192#9785352, @fgiunchedi wrote:

In T350192#9785308, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-05-10T08:30:40Z] <godog> restore SRE business hours oncall for EMEA - T350192

I noticed that batphone wasn't in the steps for sre business hours escalation, I've added back a step to route to sre emea rotation and this worked. Removing the step I believe resets the overrides when we put back the step again. I've then reset the overrides and put back folks that were there previously, which restored things as they were.

In T350192#9785308, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-05-10T08:30:40Z] <godog> restore SRE business hours oncall for EMEA - T350192

Also cc T356412: Consolidate TLS cert puppetry for ms and thanos swift frontends and @elukey since the thanos-fe work here will help with that task too

In T363660#9775097, @andrea.denisse wrote:

@fgiunchedi Good to know, thank you. Do you think we should do the syncing again to the new drive?

In T357333#9776571, @MatthewVernon wrote:

@fgiunchedi this is still causing problems (and getting emailed and poked on IRC every 4 hours is far far too noisy, we do need a more flexible policy here).

Thank you for taking a look at this @andrea.denisse @Dzahn. Filtering the targets by job ncredir (https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets?orgId=1&var-datasource=thanos&var-Filters=job%7C%3D%7Cncredir) you'll notice that the port is 3904, which is mtail and not benthos. So indeed this is T362776: replace mtail with benthos on ncredir instances and the missing bits are removing the now-obsolete ncredir job from Prometheus. (cc @Vgutierrez)

Indeed I agree that would be the root cause @colewhite pointed out. In light of the fact that (as far as I'm aware) we don't have an ETA to tweak the statsd-exporter deployment on wikikube as described in T359640; I think we should go back to the graphite/statsd metric for edits, so numbers are accurate

I've tried installing prometheus7001 today with help from @Muehlenhoff although there's no console and some pxe/tftp interaction with install7001 is suspected. I'll hold off further steps for now until VMs can be installed

Thank you for the investigation @Scott_French ! That sounds sensible to me and I'm happy to review patches for the o11y bits; on the general confd bits I'm not sure who owns the system though

Thank you all for looking into this!

lshw as requested

I have spent some time investigating this issue and I believe this is a case of https://github.com/prometheus/common/issues/598 . Specifically prometheus does reload certs from disk, however they are not used for existing connections, only new ones! If existing connections are idle for > 5 minutes then they are recycled, if that doesn't happen then existing (possibly expired) certificates are used.