Deploying Grafana Loki for

What we are setting up and why

In today's world, where applications and services generate huge volumes of data, centralized logging becomes not just a convenience, but a necessity. We will be deploying Grafana Loki — a horizontally scalable, highly available, multi-tenant log aggregation system developed by Grafana Labs. Loki differs from other logging systems in that it indexes only log metadata (labels), while storing the logs themselves in a compressed form. This makes it very efficient in terms of resource usage and storage cost.

As part of this tutorial, we will set up a complete stack for centralized collection, storage, and access to logs:

• Grafana Loki: The core of the system, which accepts logs, indexes them by labels, and stores them.
• Promtail: An agent installed on each server where logs are generated. It monitors specified log files, applies labels to them, and sends them to Loki.
• Nginx: Will act as a reverse proxy for Loki, providing secure access via HTTPS and the ability to host Loki on standard ports.
• Systemd: We will use Systemd to manage Loki and Promtail services, ensuring their automatic startup and monitoring.

Ultimately, the reader will get a fully functional centralized logging system. This will allow collecting logs from all your applications and services on a VPS (or multiple VPSs), viewing them in a convenient Grafana interface (which can be connected to Loki), quickly finding necessary events, debugging problems, and monitoring system status. This significantly simplifies infrastructure management, especially when you have multiple services or microservices deployed on one or different servers.

Alternatives: Cloud-managed vs. Self-hosted on VPS

There are several approaches to centralized logging:

• Cloud-managed services (SaaS): Platforms such as Datadog, Splunk Cloud, Logz.io, AWS CloudWatch, Google Cloud Logging, Azure Monitor. They offer ready-made solutions with minimal setup, high availability, scalability, and rich functionality. Pros: Ease of use, no need to manage infrastructure, powerful analytical tools. Cons: High cost, especially with large log volumes; potential data privacy concerns, as logs are stored with a third-party provider; less control over the infrastructure.
• Self-hosted solutions on a dedicated server/VPS: This is the approach we are implementing with Loki. You have complete control over the entire system, from infrastructure to configuration. Pros: Full control over data and security, potentially significantly lower cost for large log volumes (paying only for VPS/dedicated), flexibility in customization for specific needs. Cons: Requires technical knowledge for deployment and maintenance, responsibility for availability and scaling lies with you.

Choosing a self-hosted solution on a VPS with Loki is particularly relevant for VPS owners, developers, solo founders, and enthusiasts who want full control over their infrastructure, optimize costs, and not depend on third-party cloud providers. Loki, thanks to its design, is an excellent choice for this task, as it is less resource-intensive compared to other systems like the ELK stack (Elasticsearch, Logstash, Kibana), especially in terms of RAM and disk space consumption due to indexing only metadata.

What VPS configuration is needed for this task

Choosing the right VPS for Grafana Loki depends on the volume of logs you plan to collect and their retention period. Loki efficiently compresses logs, but still requires sufficient disk space and moderate CPU/RAM resources for indexing and queries.

Minimum requirements for Loki and Promtail (for small projects, up to 100 GB logs per month)

• CPU: 2 cores. Loki and Promtail do not heavily load the processor in idle mode, but peak loads during indexing or complex queries may require more.
• RAM: 4 GB. Loki can be quite memory-hungry with large query volumes or long-term index storage in memory. Promtail consumes little.
• Disk: NVMe SSD from 100 GB. Disk speed is critical for writing logs and executing queries. Volume depends on log volume and retention policy. For 100 GB of logs per month and 3 months of storage, about 300 GB will be needed.
• Network: 100 Mbps. For transferring logs from Promtail to Loki and accessing the UI.

Recommended VPS plan for medium projects (up to 500 GB logs per month, multiple sources)

For more serious tasks, where you collect logs from several applications or servers, and expect up to 500 GB of logs per month, you should consider a more powerful configuration:

• CPU: 4 cores.
• RAM: 8 GB.
• Disk: NVMe SSD from 500 GB to 1 TB.
• Network: 1 Gbps.

Such a plan will provide sufficient performance for processing queries and storing a larger volume of data. You can consider a VPS with the specified characteristics for deploying your centralized logging system.

When a dedicated server is needed, not a VPS

A dedicated server becomes preferable when:

• Very large log volumes: If you plan to collect terabytes of logs per month (e.g., from a large Kubernetes cluster or many microservices).
• High query performance: Instant access to logs and execution of complex analytical queries on huge data arrays are required.
• Guaranteed resources: You need 100% guaranteed CPU, RAM, and disk resources, without the "noisy neighbor" effect sometimes found on VPS.
• Strict storage requirements: Need for RAID arrays to increase disk subsystem reliability or specific disk type requirements (e.g., high-performance HDDs for archival log storage, together with SSDs for hot data).
• Physical isolation: To comply with strict security standards and compliance (e.g., PCI DSS, HIPAA), where complete physical isolation is required.

For most individual developers, solo founders, and small teams, a VPS will be more than sufficient and more cost-effective. However, if you see that your log volume exceeds 1-2 TB per month, or VPS performance begins to suffer, transitioning to a suitable dedicated server will be justified.

VPS Location: What it affects

The choice of VPS location affects several key aspects:

• Latency: The closer the VPS is to the log sources (your other servers, application clients), the lower the latency for log transmission. This is important for operational monitoring and reducing network load.
• Legal compliance: Some jurisdictions have strict data storage requirements, including logs. Ensure that the chosen location complies with applicable data privacy laws (e.g., GDPR in Europe).
• Cost: VPS prices can vary depending on the region.
• Availability: Choose regions with good network infrastructure and high data center availability.

Ideally, choose a location that is geographically close to your main log-generating servers to minimize network latency and ensure fast data transfer.

Server preparation

Server Preparation

Before proceeding with the installation of Loki and Promtail, you need to perform basic setup of your fresh VPS. This will enhance security and usability.

It is assumed that you are using Ubuntu Server 22.04 LTS or 24.04 LTS distribution, which is a popular choice for VPS.

1. SSH Connection and System Update

Connect to your server via SSH using the credentials provided by your provider. This is usually the root user and password.

ssh root@YOUR_IP_ADDRESS

After successful login, update the package list and installed packages to their latest versions:

sudo apt update && sudo apt upgrade -y

2. Creating a New User with Sudo Privileges

Working as the root user is insecure. Let's create a new user and grant them sudo privileges.

adduser lokiadmin
usermod -aG sudo lokiadmin

Now, exit the root session and log in as the new user:

exit
ssh lokiadmin@YOUR_IP_ADDRESS

3. Configuring SSH Key Authentication

To enhance security, it is recommended to use SSH keys instead of passwords. If you don't have keys yet, generate them on your local machine:

ssh-keygen -t rsa -b 4096

Copy the public key to the server (replace lokiadmin and YOUR_IP_ADDRESS):

ssh-copy-id lokiadmin@YOUR_IP_ADDRESS

Now you can disable password authentication for SSH. Edit the file /etc/ssh/sshd_config:

sudo nano /etc/ssh/sshd_config

Find and change the following lines:

#PasswordAuthentication yes
PasswordAuthentication no
#PermitRootLogin yes
PermitRootLogin no

Save changes (Ctrl+O, Enter) and exit (Ctrl+X). Restart the SSH service:

sudo systemctl restart sshd

After this, ensure that you can log in with your SSH key before closing the current session.

4. Firewall Configuration (UFW)

UFW (Uncomplicated Firewall) is a convenient wrapper for iptables. Install and configure it:

sudo apt install ufw -y
sudo ufw allow OpenSSH
sudo ufw allow http
sudo ufw allow https
sudo ufw enable
sudo ufw status

Check the status. Ports 22 (SSH), 80 (HTTP), and 443 (HTTPS) should be allowed. We will add a port for Loki later.

5. Installing Fail2Ban

Fail2Ban helps protect against brute-force attacks by blocking IP addresses that make too many failed login attempts.

sudo apt install fail2ban -y
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Fail2Ban protects SSH by default. You can configure it by editing the file /etc/fail2ban/jail.local (create it if it doesn't exist) and adding your rules.

6. Installing Basic Utilities

Install several useful utilities that will come in handy during setup and debugging:

sudo apt install curl wget git nano htop unzip -y

Your server is now ready for deploying Grafana Loki and Promtail.

Software Installation — Step-by-step

We will install Loki and Promtail by downloading binary files from official GitHub repositories and configuring them as Systemd services. Nginx will be installed from the APT repository.

The actual software versions for 2026 will be conditional, but we will use the latest stable releases available at the time of generation.

Installing Grafana Loki

Let's create directories for Loki and download the current binary file.

# 1. Create directories for Loki
sudo mkdir -p /etc/loki
sudo mkdir -p /var/lib/loki
sudo chown -R lokiadmin:lokiadmin /var/lib/loki # Grant write permissions to the current user

# 2. Change to a temporary directory
cd /tmp

# 3. Download the latest stable version of Loki (e.g., v3.1.0 for 2026)
# Always check the latest version on GitHub: https://github.com/grafana/loki/releases
LOKI_VERSION="3.1.0" # Specify the current version
wget "https://github.com/grafana/loki/releases/download/v${LOKI_VERSION}/loki-linux-amd64.zip"

# 4. Unzip the archive
unzip "loki-linux-amd64.zip"

# 5. Move the binary file to /usr/local/bin
sudo mv loki-linux-amd64 /usr/local/bin/loki

# 6. Ensure the file is executable
sudo chmod a+x /usr/local/bin/loki

# 7. Check the version
loki --version

Now, let's create a loki user, under whom the service will run, to ensure the principle of least privilege.

# 1. Create a system user loki without a home directory and shell
sudo useradd --system --no-create-home --shell /bin/false loki

# 2. Transfer ownership of the Loki data directory to the loki user
sudo chown -R loki:loki /var/lib/loki

Installing Promtail

Similar to Loki, let's download and install Promtail.

# 1. Create a directory for Promtail
sudo mkdir -p /etc/promtail

# 2. Change to a temporary directory
cd /tmp

# 3. Download the latest stable version of Promtail (e.g., v3.1.0 for 2026)
# Always check the latest version on GitHub: https://github.com/grafana/loki/releases
PROMTAIL_VERSION="3.1.0" # Specify the current version
wget "https://github.com/grafana/loki/releases/download/v${PROMTAIL_VERSION}/promtail-linux-amd64.zip"

# 4. Unzip the archive
unzip "promtail-linux-amd64.zip"

# 5. Move the binary file to /usr/local/bin
sudo mv promtail-linux-amd64 /usr/local/bin/promtail

# 6. Ensure the file is executable
sudo chmod a+x /usr/local/bin/promtail

# 7. Check the version
promtail --version

Let's create a promtail user to run the service.

# 1. Create a system user promtail without a home directory and shell
sudo useradd --system --no-create-home --shell /bin/false promtail

Installing Nginx and Certbot

Nginx will act as a reverse proxy and provide HTTPS access to Loki. Certbot will help automate the issuance and renewal of TLS certificates from Let's Encrypt.

# 1. Install Nginx from Ubuntu repositories
sudo apt install nginx -y

# 2. Start Nginx and enable it to start on boot
sudo systemctl start nginx
sudo systemctl enable nginx

# 3. Install Certbot and the Nginx plugin
sudo apt install certbot python3-certbot-nginx -y

At this stage, the main components are installed. Now let's proceed to their configuration.

Backups and Maintenance

Backup and regular maintenance are critically important for any system, especially for a logging system where data loss can be catastrophic.

What to Back Up

For Loki, the main data to back up is:

• Configuration Files: /etc/loki/config.yaml, /etc/promtail/config.yaml, Nginx configuration /etc/nginx/sites-available/loki.conf, and Let's Encrypt certificates /etc/letsencrypt/.
• Loki Data: Directories specified in storage_config (/var/lib/loki/chunks, /var/lib/loki/boltdb-shipper-active, /var/lib/loki/boltdb-shipper-cache). This data contains the logs themselves and their indexes.
• Promtail Positions: The file /var/lib/promtail/positions.yaml. It tracks how far Promtail has read each log file. If lost, Promtail may start reading logs from the beginning after recovery, leading to duplication.

Simple Auto-Backup Script

We will create a simple script that will archive key files and directories. For storing backups, it is recommended to use external storage, not the same VPS.

sudo nano /usr/local/bin/backup_loki.sh

Paste the following content. Replace /path/to/your/backup/destination with the actual path to your backup directory (e.g., a mounted S3 bucket or NFS resource).

#!/bin/bash

# Directory for temporary backups
BACKUP_DIR="/tmp/loki_backup_$(date +%Y%m%d%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Directory for storing final archives
FINAL_BACKUP_DESTINATION="/path/to/your/backup/destination" # REPLACE THIS PATH
mkdir -p "$FINAL_BACKUP_DESTINATION"

# Files and directories for backup
CONFIG_FILES="/etc/loki/config.yaml /etc/promtail/config.yaml /etc/nginx/sites-available/loki.conf"
LETSENCRYPT_DIR="/etc/letsencrypt"
LOKI_DATA_DIR="/var/lib/loki"
PROMTAIL_POSITIONS="/var/lib/promtail/positions.yaml"

# Copy configuration files and certificates
cp -r $CONFIG_FILES "$BACKUP_DIR/"
cp -r $LETSENCRYPT_DIR "$BACKUP_DIR/"
cp $PROMTAIL_POSITIONS "$BACKUP_DIR/"

# Stop Loki and Promtail for consistent data backup
# IN PRODUCTION, consider more complex methods for hot backup or replication
sudo systemctl stop loki
sudo systemctl stop promtail

# Copy Loki data (this can take a long time for large volumes)
# Use rsync for incremental copying
sudo rsync -az "$LOKI_DATA_DIR" "$BACKUP_DIR/"

# Start Loki and Promtail back up
sudo systemctl start loki
sudo systemctl start promtail

# Create archive
ARCHIVE_NAME="loki_backup_$(date +%Y%m%d%H%M%S).tar.gz"
tar -czf "$FINAL_BACKUP_DESTINATION/$ARCHIVE_NAME" -C "$BACKUP_DIR" .

# Delete temporary files
rm -rf "$BACKUP_DIR"

# Clean up old backups (e.g., keep for 7 days)
find "$FINAL_BACKUP_DESTINATION" -name "loki_backup_*.tar.gz" -type f -mtime +7 -delete

echo "Loki backup completed: $FINAL_BACKUP_DESTINATION/$ARCHIVE_NAME"

Make the script executable:

sudo chmod +x /usr/local/bin/backup_loki.sh

Where to Store Backups

Never store backups on the same server as the original data. If the server fails, you will lose both your data and your backups.

• External S3-compatible object storage: The most common and reliable option. Use services like AWS S3, DigitalOcean Spaces, Backblaze B2, or MinIO (for self-hosted). Utilities like s3cmd or rclone can be used for uploading.
• Separate VPS/Dedicated server: Allocate a separate server specifically for storing backups. You can use rsync over SSH to copy archives.
• NFS storage: If you have centralized network storage.

For integration with S3-like storage, you can install s3cmd or rclone and add an upload command to your backup script.

# Example of adding to backup_loki.sh script for S3 (after tar.gz creation)
# sudo apt install rclone -y
# rclone config # Configure rclone for your S3 provider
# rclone copy "$FINAL_BACKUP_DESTINATION/$ARCHIVE_NAME" "your_s3_remote:your-bucket-name/loki-backups/"

Automating Backups with Cron

Add the script to Cron's schedule for daily execution:

sudo crontab -e

Add the following line to execute the script daily at 03:00 AM:

0 3 * * * /usr/local/bin/backup_loki.sh >> /var/log/loki_backup.log 2>&1

Updates: Rolling vs. Maintenance Window

• Rolling updates (for a cluster): If Loki is deployed in a cluster (which we did not do in this tutorial, as it's a VPS), you can update nodes one by one to maintain system availability.
• Maintenance window (for a single VPS): For a single VPS, as in our case, Loki and Promtail updates will require a short "maintenance window" during which services will be stopped.
  1. Stop Promtail and Loki.
  2. Download new binary files.
  3. Replace old binaries with new ones.
  4. Start Loki, then Promtail.
  5. Check logs and functionality.

  It is recommended to perform updates during off-peak hours and always have a fresh backup.

Troubleshooting + FAQ

This section collects common problems and questions that may arise during the deployment and operation of Grafana Loki.

Loki or Promtail does not start, or is inactive. What to do?

If the Loki or Promtail service does not start or its status failed, the first thing to check is the Systemd logs. Use the commands:

sudo systemctl status loki.service
sudo journalctl -u loki.service -f

Similarly for Promtail: sudo systemctl status promtail.service and sudo journalctl -u promtail.service -f. Logs often contain information about configuration errors, file access issues, or occupied ports. Make sure that the config.yaml configuration files are syntactically correct (use an online YAML validator) and that the loki/promtail user has read permissions for configuration files and write permissions for data directories.

I don't see logs in Loki UI (via Grafana or curl).

If logs are not being received, check the following:

1. Promtail Status: Make sure Promtail is running and operating without errors (sudo systemctl status promtail).
2. Promtail Logs: Check Promtail logs (sudo journalctl -u promtail.service -f) for errors when sending logs to Loki. Common errors: connection refused (Loki is not available), 400 Bad Request (problem with log format or labels).
3. Promtail Configuration: Make sure that the Loki url (http://localhost:3100/loki/api/v1/push) is correctly specified in /etc/promtail/config.yaml and that the paths to log files (__path__) are correct and readable by the promtail user.
4. Loki Status: Make sure Loki is running and accessible on port 3100.
5. Firewall: Check that port 3100 is open if Loki and Promtail are on different servers or Promtail is trying to connect to Loki via a public IP.

Nginx returns a 502 Bad Gateway or 404 Not Found error.

A 502 Bad Gateway error usually means that Nginx cannot connect to the backend (in this case, Loki). Check:

1. Loki Status: Make sure Loki is running and listening on localhost:3100.
2. Nginx Logs: Check /var/log/nginx/error.log. It will precisely indicate the cause of the error, for example, connection refused.
3. Nginx Configuration: Check the file /etc/nginx/sites-available/loki.conf. Make sure that proxy_pass http://localhost:3100; is correctly specified.

A 404 Not Found error may indicate an incorrect location / configuration in Nginx or that Loki does not have such a path.

What is the minimum VPS configuration suitable for Grafana Loki?

For small projects, such as a personal blog, a small SaaS application, or several microservices generating up to 100 GB of logs per month, a VPS with 2 CPU cores, 4 GB RAM, and 100-200 GB NVMe SSD will be minimally suitable. Disk speed is more critical here than raw capacity, as Loki actively works with the disk. Ensure that the chosen tariff plan offers a sufficiently fast SSD for efficient operation with indexes and log chunks.

What to choose — VPS or dedicated for this task?

The choice between a VPS and a dedicated server depends on the scale of your task. For most individual developers, small teams, and projects with log volumes up to 1-2 TB per month, a VPS will be the optimal choice in terms of cost and ease of management. Loki is designed to be lightweight, making it ideal for a VPS. A dedicated server becomes necessary when maximum performance, guaranteed resources without overselling, very large log volumes (terabytes per day), or specific hardware requirements (e.g., RAID arrays for disks, high-performance network cards) are needed. If you are just starting, always choose a VPS, and then scale up to a dedicated server if a real need arises.

How to ensure Loki's security?

In this tutorial, we disabled authentication (auth_enabled: false) for simplicity, but this is unacceptable for production. It is recommended:

1. Basic Auth via Nginx: Nginx can provide basic HTTP authentication before proxying requests to Loki.
2. OIDC/OAuth2: If you use Grafana, it supports integration with OIDC/OAuth2, which allows for centralized authentication.
3. TLS/HTTPS: Ensure that all traffic to Loki goes over HTTPS (which we configured with Nginx and Certbot).
4. Firewall: Restrict access to Loki ports only for necessary IP addresses.
5. Principle of least privilege: Loki and Promtail run under separate system users with limited rights.

How to connect Grafana to Loki?

After deploying Loki, you can easily connect it to Grafana. To do this:

1. Install Grafana (if you haven't already).
2. In Grafana, go to Configuration -> Data Sources.
3. Click Add data source and select Loki.
4. In the URL field, specify the address of your Loki server, for example, https://your.domain.com.
5. If you are using authentication (e.g., Basic Auth via Nginx), provide the corresponding credentials.
6. Click Save & Test.

After this, you will be able to use Loki as a data source in Grafana Explore to query logs using LogQL and create dashboards.

Conclusion and Next Steps

Congratulations! You have successfully deployed and configured a powerful and cost-effective centralized logging system on your VPS using Grafana Loki, Promtail, and Nginx. Now you have full control over your logs, the ability to quickly search and analyze events, which significantly simplifies debugging and monitoring of your applications and server.

Next steps for developing your system:

• Grafana Integration: Connect your Loki server to an existing or new Grafana installation. This will allow you to leverage Grafana's powerful visualization capabilities, create dashboards, alerts, and perform deeper log analysis using the LogQL query language.
• Loki Monitoring: Set up monitoring for Loki itself using Prometheus and Grafana. Loki provides metrics in Prometheus format, which can be collected and visualized to track its performance and status.
• Scaling and High Availability: If your log volume significantly increases, consider scaling Loki by adding additional instances, using object storage (S3-compatible) instead of a filesystem for log chunks, and distributed storage for indexes (e.g., Consul or Etcd).
• Expanding Log Collection: Configure Promtail to collect logs from all your applications, including Docker containers (if you use them), as well as from other servers, if you have any.