plesiosaur.net ‘s mastodon ran out of space sometime on July 2, 2023. While I look for the underlying cause to see if it is something other than neglecting (optimistically) to turn on retention policies for remote content, it may be useful for others to see this diagnostic trick I am running.tootctl is a tool used to control mastodon from the command line. I edited crontab to log the output of tootctl media usage to a file.
Crucially, if you are new to running a mastodon instance and to #docker and #linux, you’ll want to have edited your .bashrc profile first to ensure that your environment is set to production. This will help in the next step.
echo "export RAILS_ENV=production" >> ~/.bashrcNext, for convenience I am going to log the output to the host VPS and not within the docker container.
This will require running a process through docker exec . This is the power of a container over a VM, a process can be initiated and interacted with from the host to the guest.
The full command is going to look like this: docker exec -it <name> tootctl media usage .
And because I like things to be organized, I’ll grab the date and time and put it directly before this in crontab. If you aim to copy and paste this for your own crontab, please note that the paths and container name will need to be corrected to match your environment.
0 */6 * * * date >> /../mastodon_media_usage.log ; docker exec -it <name> tootctl >> /../mastodon_media_usage.logSince it took about two months to exhaust the space on disk, I’ll monitor over the coming month to see how quickly (or if) the instance exhausts the additional space I added to the VPS. I did also turn on the remote content retention policy and that may help with the current disk utilization.
Also note that if you see a lot of ‘port enabled|blocking’ state messages in your logs, that can be caused by docker running out of diskspace to maintain its virtual switch. They are spanning tree protocol messages
Resources:
- Display Diskspace using tootctl https://sleeplessbeastie.eu/2022/12/08/how-to-display-disk-space-consumed-by-mastodon/
- Using tootctl https://docs.joinmastodon.org/admin/tootctl/
- Troubleshooting mastodon’s diskspace in a containerized environment https://www.reddit.com/r/Mastodon/comments/11zky7d/ubuntu_1804_full_disk_crushed_my_mastodon/
- Where to see logs in dockerized mastodon: https://docs.joinmastodon.org/admin/troubleshooting/
Thank you for reading. Please let me know if there are any clarifications I can make or further questions I can answer either down in the comments, on LinkedIn, or hit me up on Mastodon.