2018/06/22

Friday, June 22, 2018 (#173)

Thursday Friday Saturday prev .. today .. next 2024-11-29 (Fri) 11:19 EST Everything seems to be working ok... (details of last incident.) There are no known issues. This isn't the first time this has happened... outgoing toots are being delayed by something like 1.5-2 hours. Last time, we upped the number of threads from 15 to 25 (according to my memory; apparently we didn't document it), and that seemed to fix the blockage. So this time, I'm making notes. Editing /etc/systemd/system/mastodon-sidekiq.service: changing DB_POOL=5 to DB_POOL=20 changing sidekiq -c 25 to sidekiq -c 40 ...and rebooted (which wasn't strictly necessary, but the system was asking for one because of a kernel upgrade some days ago). ...and then it turned out I had forgotten to save my changes before rebooting, so I had to do this: root@tootcat2:~# systemctl restart mastodon-sidekiq.service Warning: mastodon-sidekiq.service changed on disk. Run 'systemctl daemon-reload' to reload units. root@tootcat2:~# systemctl daemon-reload After finding this explanation, I understood two things: DB_POOL should be the same as the sidekiq -c value I actually have plenty of CPU headroom, and can boost this quite a bit more. I first tried 40, and that helped a bit -- but the "busy" queue was still getting stuck around 3500. So then I bumped it up to 100, and within a minute or two the queue was down under 100. Also, do things in this order: root@tootcat2:~# systemctl daemon-reload root@tootcat2:~# systemctl restart mastodon-sidekiq.service