We'll use wdqs2023 to compare against wdqs1023 (same hardware). 1023 is scholarly-articles so that's what we'll want to load 2023 with.

Actually we'll use wdqs1021 since we're not confident NFS will work seamlessly between eqiad and codfw (we use nsf to procure the dumps that the data-reload is run from).

Gearing up to kick off a data reload on wdqs1021 after a quick patch to enable NFS on it

Change #1026668 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: enable nfs data reloads on wdqs1021

https://gerrit.wikimedia.org/r/1026668

I'm realizing I don't remember enough about how we load specific graph splits (scholarly vs main). But it's possible we won't need the above nfs patch if our previous process was to manually download the relevant dump file.

@dcausse We want to do a data reload of the scholarly graph for a WDQS host like wdqs2023. What's the process for that again?

@RKemper I think that's captured in P54284 . If you need to get a copy of the files, there's a pointer in T350106#9381611 for how one might go about copying from HDFS to the local filesystem and then there's other stuff in the rest of the ticket about the data transfer. I kept a copy of the files at stat1006:/home/dr0ptp4kt/gzips/nt_wd_schol so those should be ready to be copied over if that helps at all.

Another thing that can be nice for figuring out stuff later is to add some timing and a simple log file. A command like the following was helpful when I was trying this out on the gaming-class desktop (you may not need this if your tmux session lets you scroll back really far, but it's kind of nice for tailing even without tmux).

date | tee loadData.log; time ./loadData.sh -n wdq -d /mnt/firehose/split_0/nt_wd_schol -s 0 -e 0 2>&1 | tee -a loadData.log; time ./loadData.sh -n wdq -d /mnt/firehose/split_0/nt_wd_schol 2>&1 | tee -a loadData.log

Change #1027001 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: switch wdqs2023 to graph split host

https://gerrit.wikimedia.org/r/1027001

Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:27:31Z] <ryankemper> T362920 [wdqs] Depooled wdqs2023 in preparation to switch it to a graph split host

Change #1027001 merged by Ryan Kemper:

[operations/puppet@production] wdqs: switch wdqs2023 to graph split host

https://gerrit.wikimedia.org/r/1027001

Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:38:44Z] <ryankemper@cumin2002> START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 on wdqs2023.codfw.wmnet with reason: T362920

Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:38:48Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on wdqs2023.codfw.wmnet with reason: T362920

Change #1026668 abandoned by Ryan Kemper:

[operations/puppet@production] wdqs: enable nfs data reloads on wdqs1021

Reason:

don't need nfs for graph split reload

https://gerrit.wikimedia.org/r/1026668

Mirroring comment in T359062#9775908:

In T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors) we saw that this took about 3702 minutes, or about 2.57 days, for the scholarly article entity with the CPU governor change (described in T336443#9726600 ) alone on wdqs2023.

The count matches T359062#9695544.

select (count(*) as ?ct)
where {?s ?p ?o}

7643858078

@dr0ptp4kt

we saw that this took about 3702 minutes, or about 2.57 hours

Typo you'll want to fix here and in the original: 2.57 days

Kicked off the second run like so:

(1) vi /etc/wdqs/RWStore.wikidata.properties (added a trailing 0 to buffersize)

(2) date | tee /home/ryankemper/loadData2.log; time sudo /home/ryankemper/loadData.sh -n wdq -d /srv/T362920/nt_wd_schol -s 0 -e 0 2>&1 | tee -a /home/ryankemper/loadData2.log; time sudo /home/ryankemper/loadData.sh -n wdq -d /srv/T362920/nt_wd_schol 2>&1 | tee -a /home/ryankemper/loadData2.log; echo -en "\nrun finished at " | tee /home/ryankemper/loadData2.log; date | tee /home/ryankemper/loadData2.log

In T362920#9776418, @RKemper wrote:

@dr0ptp4kt

we saw that this took about 3702 minutes, or about 2.57 hours

Typo you'll want to fix here and in the original: 2.57 days

I think this is what is referred to as wishful thinking! Okay, updated the comment in the other ticket's comment and in the comment up above.

Mirroring comment in T359062#9783010:

And for the second run in T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors) we saw that this took about 3089 minutes, or about 2.15 days, for the scholarly article entity graph with the CPU governor change (described in T336443#9726600 ) plus the bufferCapacity at 1000000 on wdqs2023.