Maximum performance at minimal cost. Start your server in seconds now!
    Vijona

    Managing System Logs with Fluent Bit and OpenSearch

    Managing system logs is crucial for maintaining performance, troubleshooting issues, and understanding system behavior. System logs, typically stored in /var/log, provide valuable insights into the operation of your server. This tutorial will guide you through installing Fluent Bit on a ccloud³ VM, configuring it to collect system logs from /var/log, and sending them to OpenSearch for comprehensive analysis.

    Prerequisites

    Before you start, ensure you have the following:

    • A ccloud³ VM.
    • A OpenSearch Cluster.

    Step 1 – Installing Fluent Bit

    Fluent Bit is an open-source and lightweight log processor and forwarder. It is designed to collect data and logs from various sources, process or transform them, and then forward them to different destinations.

    Fluent Bit can be installed on multiple platforms. To install it on Ubuntu/Debian/Redhat/CentOS, run the following command on your ccloud³ VM terminal:

    Copy Code 
    

    curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

    To more information on Fluent Bit installation, refer to Fluent Bit documentation.

    Step 2 – Configuring Fluent Bit to Send Logs to OpenSearch

    By default, Fluent Bit configuration files are located in /etc/fluent-bit/. To forward logs to OpenSearch, you’ll need to modify the fluent-bit.conf file.

    Fluent Bit Inputs

    Fluent Bit offers a variety of input plugins that enable it to collect log and event data from different sources. Since we will be sending logs from log files, we will be using the tail input plugin.

    Update the fluent-bit.conf file as follows:

    Copy Code 
    

    [INPUT]
    name tail
    Path /var/log/auth.log,/var/log/syslog,/var/log/journal/*.log

    Change the Path to the desired log path. For more information on input plugins, refer to Fluent Bit Input Plugins.

    Fluent Bit Outputs

    Similar to input plugins, Fluent Bit provides an output plugin that sends collected and processed logs to different destinations. Since we are sending logs to OpenSearch, let’s make use of the OpenSearch Output Plugin.

    Copy Code 
    

    [OUTPUT]
    Name  opensearch
    Match *
    Host 
    Port 25060
    HTTP_User doadmin 
    HTTP_Passwd  
    Index ubuntu
    tls On
    Suppress_Type_Name On

    Replace <OpenSearch_Host> with your OpenSearch server’s hostname and <OpenSearch_Password> with your OpenSearch password.

    Starting Fluent Bit Service

    Once the configurations are set, start the Fluent Bit service by executing the following commands:

    Copy Code 
    

    systemctl enable fluent-bit.service
systemctl start fluent-bit.service
systemctl status fluent-bit.service

    Troubleshooting

    Check Connectivity

    You can verify that Logstash can connect to OpenSearch by testing connectivity:

    Copy Code 
    

    curl -u your_username:your_password -X GET "https://your-opensearch-server:25060/_cat/indices?v"

    In this command, replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username, and your_password with your OpenSearch credentials.

    Data Ingestion

    Ensure that data is properly indexed in OpenSearch:

    Copy Code 
    

    curl -u your_username:your_password -X GET "http://your-opensearch-server:25060/<your-index-name>/_search?pretty"

    Replace your-OpenSearch-server with your OpenSearch server’s hostname, your_username and your_password with your OpenSearch credentials, and your-index-name with the index name.

    Firewall and Network Configuration

    Ensure firewall rules and network settings allow traffic between Logstash and OpenSearch on port.

    Check Fluent Bit Logs

    By default, logs are written to the system log.

    Copy Code 
    

    sudo journalctl -u fluent-bit

    Validate Configuration

    Ensure the configuration files are syntactically correct.

    Copy Code 
    

    /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run

    Conclusion

    In this tutorial, we covered the essential steps to set up Fluent Bit for managing system logs, including installation, configuration, and forwarding logs to OpenSearch.

    A recap of what we’ve done:

    • Installation: We installed Fluent Bit on a ccloud³ VM using a simple curl command.
    • Configuration: We configured Fluent Bit to collect system logs from /var/log using the tail input plugin and send them to OpenSearch using the OpenSearch output plugin.
    • Service Management: We enabled and started the Fluent Bit service to ensure continuous log collection and forwarding.
    • Troubleshooting: We addressed common troubleshooting steps, including verifying connectivity, checking data ingestion, and reviewing Fluent Bit logs.

    With these steps completed, Fluent Bit will efficiently collect and forward your system logs to OpenSearch, allowing you to leverage powerful search and analysis tools to gain insights into your server’s performance and behavior.

    Create a Free Account

    Register now and get access to our Cloud Services.

    Try now

    Posts you might be interested in: