Search

On or before date

go

2020-02-14 §
20:18	<andrewbogott>	adding profile::rsyslog::kafka_shipper::kafka_brokers: [] to project-wide puppet setting	[ores]
20:16	<andrewbogott>	adding profile::ores::logstash_host: localhost to project-wide puppet config	[ores]
2020-02-12 §
15:28	<halfak>	staging ores-wmflabs-deploy:4f0a14e	[ores]
2019-10-10 §
19:20	<halfak>	quiet icinga-wm in #wikimedia-ai	[ores]
16:25	<halfak>	unquiet icinga-wm in #wikimedia-ai	[ores]
2019-10-08 §
14:07	<halfak>	Restarting nginx on ores-lb-03	[ores]
13:49	<halfak>	Temporarily stopped nginx on ores-lb-03	[ores]
13:24	<halfak>	running FLUSHDB on ores-redis-02	[ores]
13:13	<halfak>	restarting celery workers on ores-worker-01,02,03,04	[ores]
2019-09-25 §
18:13	<halfak>	restarted redis services on ores-redis-02 (again)	[ores]
18:12	<halfak>	deleting AOF file on ores-redis-02	[ores]
18:03	<halfak>	restarted redis services on ores-redis-02	[ores]
2019-09-13 §
21:52	<halfak>	switched ores-lb-03 to point to ores-web-04/05/06	[ores]
21:17	<andrewbogott>	moving ores-redis-02 to cloudvirt1030	[ores]
21:06	<halfak>	restarting redis services on ores-redis-02	[ores]
20:53	<halfak>	ran 'sudo service celery-ores-worker restart' on celery-worker-02	[ores]
2019-09-12 §
21:20	<bd808>	Forcing puppet run on ores-web-04.ores.eqiad.wmflabs to see if I can fix ssh access for halfak	[ores]
2019-09-10 §
15:27	<halfak>	rebuilt ores-worker-03 with role::labs::ores::worker	[ores]
14:26	<halfak>	restarting ores-uwsgi service on ores-web-01	[ores]
2019-09-09 §
22:08	<halfak>	deleting ores-worker-03 because it didn't work	[ores]
21:50	<halfak>	configured ores-worker-03 with role::labs::ores::worker	[ores]
2019-09-06 §
17:18	<halfak>	staging ores-wmflabs-deploy:f76823b	[ores]
16:46	<halfak>	ran apt-get update/upgrade on ores-web-01	[ores]
15:43	<halfak>	manually running puppet on ores-web-02 and 03	[ores]
15:40	<halfak>	manually running puppet on ores-web-01	[ores]
15:39	<halfak>	lowering web workers per node to 36 https://wikitech.wikimedia.org/w/index.php?title=Hiera:Ores&diff=1836902&oldid=1826595	[ores]
15:29	<halfak>	manually disabling uwsgi statsd on ores-web-01	[ores]
2019-09-04 §
20:20	<halfak>	ran "flushall" on both redis instances (ports 6379 and 6380)	[ores]
20:09	<halfak>	restarting all labs VPN nodes.	[ores]
15:33	<halfak>	ran "sudo service nginx restart" on ores-lb-03	[ores]
2019-06-14 §
22:06	<halfak>	ran "sudo service nagios-nrpe-server restart" on ores-web-01	[ores]
2019-06-11 §
18:50	<halfak>	deploying ores-wmflabs-deploy:079f18c	[ores]
18:48	<halfak>	staging ores-wmflabs-deploy:079f18c	[ores]
17:19	<halfak>	deploying ores-wmflabs-deploy:989c436	[ores]
17:16	<halfak>	staging ores-wmflabs-deploy:989c436	[ores]
2019-06-06 §
21:42	<halfak>	staging ores-wmflabs-deploy:989c436	[ores]
2019-04-15 §
08:12	<Amir1>	f63fddf going staging	[ores]
2019-03-18 §
14:05	<halfak>	deploying ores-wmflabs-deploy:9a0e589	[ores]
14:02	<halfak>	staging ores-wmflabs-deploy:9a0e589	[ores]
2019-02-21 §
15:29	<halfak>	staging ores-wmflabs-deploy:ea153f6	[ores]
2019-02-08 §
16:02	<halfak>	staging ores-wmflabs-deploy:5bd6b61	[ores]
2019-01-31 §
12:05	<arturo>	VM instances ores-puppetmaster-01,ores-sentinel-01, were stopped briefly due to issue in hypervisor (T215012)	[ores]
2019-01-07 §
21:13	<halfak>	deploying ores-wmflabs-deploy:5634d1	[ores]
21:08	<halfak>	staging ores-wmflabs-deploy:5634d1	[ores]
2019-01-04 §
18:17	<halfak>	deploying ores-wmflabs-deploy:f3dbd2	[ores]
18:13	<halfak>	staging ores-wmflabs-deploy:f3dbd2	[ores]
16:06	<halfak>	ran "sudo service uwsgi-ores restart" on ores-staging-01	[ores]
15:39	<halfak>	staging ores-wmflabs-deploy:7a3bdf	[ores]
2018-11-07 §
19:05	<Amir1>	ores:f2f085d is going prod	[ores]
2018-11-03 §
13:53	<Amir1>	restarting ores-redis-02 under pressure	[ores]