Effective Web Server Log Management with Logstash and OpenSearch
Effective web server log management is crucial for maintaining your website’s performance, troubleshooting issues, and gaining insights into user behavior. Apache is one of the most popular web servers. It generates access and error logs that contain valuable information.
In this tutorial, we will guide you through installing Logstash on a Droplet, configuring it to collect your Apache logs, and sending them to Managed OpenSearch for analysis.
Prerequisites
- Droplet/s with Apache Webserver installed.
- Managed OpenSearch Cluster
Step 1 – Installing Logstash
Logstash can be installed using the binary files OR via the package repositories. For easier management and updates, using package repositories is generally recommended.
In this section, we’ll guide you through installing Logstash on your Droplet using both APT and YUM package managers.
Let’s identify the OS:
cat /etc/os-release
For APT-Based Systems (Ubuntu/Debian)
Download and install the Public Signing Key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
You may need to install the apt-transport-https package on Debian before proceeding:
sudo apt-get install apt-transport-https
Save the repository definition to /etc/apt/sources.list.d/elastic-8.x.list:
echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
Use the echo method described above to add the Logstash repository. Do not use add-apt-repository as it will add a deb-src entry as well, but we do not provide a source package. If you have added the deb-src entry, you will see an error like the following:
Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)
Just delete the deb-src entry from the /etc/apt/sources.list file and the installation should work as expected.
Run sudo apt-get update and install Logstash:
sudo apt-get update && sudo apt-get install logstash
For YUM-Based Systems (CentOS/RHEL)
Download and install the public signing key:
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Add the following in your /etc/yum.repos.d/logstash.repo file. You can make use of ‘tee’ to update and create the file.
sudo tee /etc/yum.repos.d/logstash.repo > /dev/null <
Install Logstash:
sudo yum install logstash
For further information, please refer to the Installing Logstash guide.
Step 2 – Configuring Logstash to Send Logs to OpenSearch
A Logstash pipeline consists of three main stages: input, filter, and output. Logstash pipelines make use of plugins. You can make use of community plugins or create your own.
Input
This stage collects data from various sources. Logstash supports numerous input plugins to handle data sources like log files, databases, message queues, and cloud services.
Filter
This stage processes and transforms the data collected in the input stage. Filters can modify, enrich, and structure the data to make it more useful and easier to analyze.
Output
This stage sends the processed data to a destination. Destinations can include databases, files, and data stores like OpenSearch.
Step 3 – Installing the Open Search Output Plugin
The OpenSearch output plugin can be installed by running the following command:
/usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch
More information can be found on the logstash-output-opensearch-plugin repository.
Creating a Logstash Pipeline
Now, let’s create a pipeline.
Create a new file in the path /etc/logstash/conf.d/ called apache_pipeline.conf, and copy the following contents.
input {
file {
path => "/var/log/apache2/access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
tags => "apache_access"
}
file {
path => "/var/log/apache2/error.log"
start_position => "beginning"
sincedb_path => "/dev/null"
tags => "apache_error"
}
}
filter {
if "apache_access" in [tags] {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
mutate {
remove_field => [ "message","[log][file][path]","[event][original]" ]
}
} else {
grok {
match => { "message" => "%{HTTPD24_ERRORLOG}" }
}
}
}
output {
if "apache_access" in [tags] {
opensearch {
hosts => "https://:25060"
user => "doadmin"
password => ""
index => "apache_access"
ssl_certificate_verification => true
}
} else {
opensearch {
hosts => "https://:25060"
user => "doadmin"
password => ""
index => "apache_error"
ssl_certificate_verification => true
}
}
}
Replace <OpenSearch-Hostname> with your OpenSearch server’s hostname and <your_password> with your OpenSearch password.
Understanding the Configuration
INPUT
This section is used to configure a source for the events. The file input plugin is used here.
path => "/var/log/apache2/access.log": Specifies the path to the Apache access log file that Logstash will read from.- Ensure that the Logstash service has access to the input path.
start_position => "beginning": Defines where Logstash should start reading the log file. “beginning” means Logstash will process the file from the start.sincedb_path => "/dev/null": Specifies the path to a sincedb file, which keeps track of the last read position.tags => "apache_access": Assigns a tag to events read from this input, useful for filtering later.
FILTER
Filters process and transform events.
Conditional Processing
if "apache_access" in [tags]: This checks if the tag apache_access exists in the event and applies the appropriate GROK filter.
Grok Filter for Apache Access Logs
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
The %{HTTPD_COMBINEDLOG} grok filter extracts fields such as IP address, timestamp, HTTP method, URI, and status code.
Mutate Filter
mutate {
remove_field => [ "message","[log][file][path]","[event][original]" ]
}
This removes unnecessary fields after parsing the logs.
Grok Filter for Apache Error Logs
grok {
match => { "message" => "%{HTTPD24_ERRORLOG}" }
}
This extracts structured fields from Apache error logs.
OUTPUT
The output plugin sends events to a particular destination.
Routing Logs to OpenSearch
if "apache_access" in [tags] {
opensearch {
hosts => "https://XXX:25060"
user => "doadmin"
password => "XXXXX"
index => "apache_access"
ssl_certificate_verification => true
}
}
hosts: Specifies the OpenSearch server.user: Username for authentication.password: Password for authentication.index: Defines the OpenSearch index where logs will be stored.ssl_certificate_verification => true: Enables SSL verification.
GROK patterns can be found at: Logstash Patterns Repository.
Step 4 – Start Logstash
Once the Pipeline is configured, start the Logstash service:
systemctl enable logstash.service
systemctl start logstash.service
systemctl status logstash.service
Step 5 – Troubleshooting
Check Connectivity
You can verify that Logstash can connect to OpenSearch by testing connectivity:
curl -u your_username:your_password -X GET "https://your-opensearch-server:25060/_cat/indices?v"
Replace with your OpenSearch server’s hostname and , with your OpenSearch credentials.
Data Ingestion
Ensure that data is properly indexed in OpenSearch:
curl -u your_username:your_password -X GET "http://your-opensearch-server:25060//_search?pretty"
Replace <your-opensearch-server> with your OpenSearch server’s hostname and <your_username>, <your_password> with your OpenSearch credentials. Similarly, replace <your-index-name> with the index name.
Firewall and Network Configuration
Ensure firewall rules and network settings allow traffic between Logstash and OpenSearch on port 25060.
Logs
The logs for Logstash can be found at:
/var/log/logstash/logstash-plain.log
For details, refer to the Troubleshooting Guide.
Conclusion
In this guide, we walked through setting up Logstash to collect and forward Apache logs to OpenSearch. Here’s a quick recap of what we covered:
- Installing Logstash: We covered how to use either APT or YUM package managers, depending on your Linux distribution, to install Logstash on your Droplet.
- Configuring it: We created and adjusted the Logstash configuration file to ensure that Apache logs are correctly parsed and sent to OpenSearch.
- Verifying in OpenSearch: We set up an index pattern in OpenSearch Dashboards to confirm that your logs are being indexed properly and are visible for analysis.
With these steps completed, you should now have a functional setup where Logstash collects Apache logs and sends them to OpenSearch.
