Problem with Prometheus in Sametime monitoring dashboard

I set up the monitoring dashboard in Sametime, using the provided JSON files; in case you don’t know how to do that, you enable it running this command

docker-compose -f docker-compose.yml -f docker-compose-monitoring.yml up -d

Today I discovered that my Prometheus container kept crashing, and in the logs I found lines like these:

corrupted block 01GQT2R33ASC29W57NF4WEZKP3: open data/01GQT2R33ASC29W57NF4WEZKP3/chunks: no such file or directory;
 corrupted block 01GQW0HNMW76WDHNY6T8CMEVQ0: open data/01GQW0HNMW76WDHNY6T8CMEVQ0/chunks: no such file or directory;
 corrupted block 01GQGDRDRKSS062T1762WACX5H: open data/01GQGDRDRKSS062T1762WACX5H/chunks: no such file or directory;
 corrupted block 01GQJBHXKJW38754FR7CZGTWZN: open data/01GQJBHXKJW38754FR7CZGTWZN/chunks: no such file or directory;
 corrupted block 01GQM9BGT78HDQXM2EZ2BP5P8S: open data/01GQM9BGT78HDQXM2EZ2BP5P8S/chunks: no such file or directory;
 corrupted block 01GQP75073Q7FRW2MEGE1T5869: open data/01GQP75073Q7FRW2MEGE1T5869/chunks: no such file or directory;
 corrupted block 01GQR4YKN8XJS7FY30VYR24VTG: open data/01GQR4YKN8XJS7FY30VYR24VTG/chunks: no such file or directory"

In fact looking at the directory structure under /tmp/prometheus, the default directory where Prometheus store files I found this:

The solution was to delete the directories that didn’t have a “chunks” subdirectory and restart the Prometheus container. The dashboard started working again.
Received a new badge
From LotusScript to ChatGpt and vice versa

Leave a Reply

Your email address will not be published / Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.