1-50 of 168 results (33ms)
2025-09-16 §
23:37 <mutante> rebooting phabricator-bullseye [devtools]
2025-07-23 §
18:39 <mutante> gitlab-1002 rm /etc/apt/sources.list.d/thirdparty-gitlab-bullseye-apt.wikimedia.org-wikimedia-bookworm-wikimedia.sources | this is a bookworm instance, so bullseye sources won't work - T400289 [devtools]
2025-06-13 §
18:13 <mutante> - deleted puppetmaster-1004 after confirming no instance uses it, only puppetmaster-1003 is used and they are the same version (T390948) [devtools]
2025-06-04 §
20:49 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T396006) [devtools]
20:49 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T396006) [devtools]
2025-04-09 §
17:20 <mutante> upgrading gitlab-ce version on gitlab-prod-1002 [devtools]
2025-04-04 §
18:28 <mutante> - deleting instance codesearch. replace by instance sourcebot1 in project codesearch [devtools]
2025-03-13 §
02:56 <mutante> gitlab-prod-1002 - ran out of disk when trying to upgrade gitlab version - apt-get clean [devtools]
2025-03-03 §
19:44 <mutante> editing /etc/mysql/mariadb.conf.d/50-server.cnf to point mariab datadir to /srv/sqldata ; systemctl restart mariadb --> fixes test Phabricator (T387619) [devtools]
2025-02-21 §
22:57 <mutante> puppetmaster-1004 - apt-get remove --purge puppetserver; run-puppet-agent [devtools]
2024-08-01 §
19:18 <mutante> gitlab-prod-1002 - moved contents of /var/log/gitlab out of the way temp, emptied it, letting wmcs-prepare-cinder-volume mount new volume into /var/log/gitlab and format it with ext4, moved old existing logs back into it T371066 [devtools]
19:11 <mutante> attaching volume 'gitlab-prod-logs' to instance 'gitlab-prod-1002' T371066 | running 'sudo wmcs-prepare-cinder-volume' manually and answering interactive questions to mount/format it (once T371573 is resolved will do this with puppet) [devtools]
19:07 <mutante> creating new 20GB volume 'gitlab-prod-logs' T371066 [devtools]
09:21 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [devtools]
09:21 <sstefanova@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase [devtools]
2024-07-24 §
22:42 <mutante> gitlab-prod-1002 out of disk again when attempting package upgrade - apt-get clean; rm /var/log/syslog.2*.gz rm /var/log/messages.2*.gz to get space [devtools]
2024-07-10 §
21:46 <mutante> gitlab-runner* instances: apt-get upgrade (upgrading gitlab-runner, exim*. downgrading lshw, pythong3-jinja2) [devtools]
21:34 <mutante> gitlab-prod-1002 apt-get upgrade (upgrading gitlab-ce, exim-*. downgrading lshw, python3-jinja2) [devtools]
2024-06-18 §
10:31 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [devtools]
10:18 <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.migrate_project_to_ovs [devtools]
2024-05-06 §
20:37 <mutante> deleting buster deployment server deploy-1004, replaced by deploy-1006 - T360964 [devtools]
2024-05-02 §
23:52 <mutante> switching puppetmaster for deploy-1006 back to local project puppetmaster; rm -rf /var/lib/puppet/ssl that still referred to puppetmaster-1001, signing new request on puppetmaster-1003 T360470 T363415 [devtools]
23:10 <mutante> replacing deploy-1004 (buster) with deploy-1006 (bullseye) as new deployment server in both repo and Horizon hiera T360964 T363415 [devtools]
2024-05-01 §
23:59 <mutante> creating instance gerrit-bullseye (T363196) [devtools]
22:45 <mutante> rebooting deploy-1006 to see if issue with starting mw-cgroup goes away - .. and it did! - it must have been the grub config from https://gerrit.wikimedia.org/r/c/operations/puppet/+/991347 but needed one reboot - T363957 [devtools]
2024-04-24 §
20:28 <mutante> changing puppetmaster of deploy-1006 to puppetmaster.cloudinfra.wmflabs.org instead of the local project one [devtools]
20:12 <mutante> - puppet run on deploy-1006 deployment_server - scap init duplicate declaration fixed after adding "profile::mediawiki::scap_client::is_master: true" to new deploy-prefix Hiera - thanks to Jaime Nuche per comment on gerrit [devtools]
19:46 <mutante> creating new prefix 'deploy' to apply needed Hiera keys to deployment hosts based on host name prefix (both deploy1004, deploy1006 and future deploy*) [devtools]
19:41 <mutante> deleting instance puppetmaster-1001 that was > 4 years old, on buster and I had shutdown a couple days ago. replaced by puppetmaster-1003 (bookworm, puppetserver) T360964 T360470 [devtools]
19:21 <mutante> gitlab-prod-1002; gitlab-runner-1003; gitlab-runner-1002 - apt-get update && apt-get upgrade [devtools]
2024-04-17 §
21:18 <mutante> - resizing puppetmaster-1003 from g3.cores1.ram2.disk20 to g3.cores2.ram4.disk20 - T360470 [devtools]
2024-04-16 §
17:09 <mutante> - deleting devtools-puppetdb1001 instance (T360964) [devtools]
16:53 <mutante> - soft rebooting puppetmaster-1003, shutting down puppetmaster-1001 [devtools]
16:38 <mutante> - can't ssh to new puppetmaster again [devtools]
16:33 <mutante> - deleting deploy-1005 - don't try deployment server in bookworm, first bullseye [devtools]
16:29 <mutante> - shutting down puppetdb instance again [devtools]
2024-04-15 §
17:52 <mutante> - added profile::labs::cindermount::srv to puppetmaster-1003 in horizon to get missing cinder volume - T360470 [devtools]
17:51 <mutante> - added Notice: /Stage[main]/Profile::Labs::Cindermount::Srv/Cinderutils::Ensure[cinder_on_srv]/Exec[prepare_cinder_volume_/srv]/returns: executed successfully [devtools]
16:57 <mutante> - puppetmaster-1003 reachable again but service fails to start and puppetserver-deploy-code fails [devtools]
16:50 <mutante> - rebooted unreachable puppetmaster-1003 - was "no route to host" - but is back now, log had a " /dev/sdb: Can't open blockdev" as well [devtools]
2024-04-12 §
17:55 <mutante> - changed both puppetmaster and puppetdb hiera setting back to puppetmaster-1001 for instance deploy-1004 [devtools]
17:49 <mutante> - deploy-1004 itself is on buster and buster and puppet 7 don't mix well - testing deployment role on bookworm, creating deploy-1005 [devtools]
17:48 <mutante> - deploy-1004 has a puppet problem when talking to new puppetmaster-1003 that goes away when switching back to puppetmaster-1001 [devtools]
2024-04-11 §
20:08 <mutante> zuul-1001 - switching to new puppetmaster-1003 in puppet.conf manually, switched project defaults in repo too [devtools]
19:58 <mutante> manually editing puppet.conf to use puppetmaster-1003 instead of -1001 because you can't switch the puppetmaster via puppet if puppet is already broken :) [devtools]
19:41 <mutante> switching gitlab-runner-1005 from puppetmaster-1001 to puppetmaster-1003 via web Hiera [devtools]
19:03 <mutante> - deleting instance contint-bullseye which was only used by me for a test before we created contint1003 in prod T334517 T361224 [devtools]
18:38 <mutante> - attempting to fix puppet run on vrts-1001 related to switching prod to cfssl for SSL cers [devtools]
18:23 <mutante> - shutting down puppetmaster-1001 on buster - should now be replaced by puppetmaster-1003 on bookworm (thanks brennen) T360964 T360470 [devtools]
18:01 <mutante> - shutting down instance devtools-puppetdb1001 - which is on buster - basically to see what breaks of complains, if anything [devtools]