Centralized logging for a dozen of devices

[worn delta]

I need to build a logging system for QA testing devices in production line. There are around 25 such devices, with 7 or so different types that all work differently (I more or less understand 2 of them). At this moment I don't know if all of them will be on network and what are their capabilities. I will need to track how long each test lasts, what's the failure rate for each of them and present them in easy for managers to understand way.

Do you have any tips on how should I structure this? I'm thinking of doing two databases: one for raw logs, other for structured results for presentation. That way I could have different structure from each device type and format them later into universal structure in a cron job.

Has anyone else build something similar? Some hindsight would help.

[elder locust]

<@&987246841693360200> please have a look, thanks.

Here is an AI assisted attempt to answer your question 🤖. Maybe it helps! In any case, a human is on the way 👍. To continue talking to the AI, you can use </chatgpt:1108714622413963314>.

erently. The goal is to have a centralized logging system that can collect and store logs from all these devices for easy monitoring and troubleshooting.

Here are the steps you can follow to build a centralized logging system for your QA testing devices:

1. Determine the logging requirements: Understand what kind of logs you need to collect from each device. Identify the log types, formats, and any specific information that needs to be captured.

2. Choose a logging framework or tool: Research and select a suitable logging framework or tool that supports your requirements. Some popular options include ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, Graylog, or Fluentd.

3. Set up a centralized log server: Create a dedicated server or use an existing one to act as the central log repository. Install the chosen logging tool on this server.

4. Configure log collection: For each device, configure its respective logging mechanism to send logs to the central log server. This can be done by modifying the device's configuration files or using an agent installed on each device.

5. Standardize log formats: Ensure that all devices follow a standardized log format so that they can be easily parsed and analyzed by the centralized logging system.

6. Implement log shipping: Set up mechanisms to ship logs from each device to the central server securely. This can be done using protocols like syslog, HTTPS, or custom APIs depending on your chosen logging tool's capabilities.

7. Define log retention policies: Determine how long you want to retain logs on the central server before archiving or deleting them based on your organization's needs and compliance requirements.

8. Configure log parsing and indexing: Use the features provided by your chosen logging tool to parse and index incoming logs for efficient searching and analysis.

9. Set up monitoring and alerts: Configure alerts based on specific log patterns or anomalies that may indicate issues with any of the devices under testing.

10. Create dashboards and visualizations: Utilize the visualization capabilities of your logging tool to create informative dashboards and reports that provide insights into the performance and behavior of the devices.

11. Test and iterate: Test the centralized logging system thoroughly to ensure it is collecting logs from all devices correctly. Iterate on the configuration and setup as needed to address any issues or improvements.

12. Train QA testers and support staff: Provide training to QA testers and support staff on how to access, search, and utilize the centralized logging system effectively for troubleshooting purposes.

Remember to regularly monitor the centralized logging system's performance, storage requirements, and make necessary adjustments as your device fleet grows or changes over time.

[mint crescent]

What kind of 'devices'? If you don't have domain knowledge, don't bother trying to create some kind of grand-architecture. Do a system for each device type or communication protocol and then when you have more knowledge and after your stakeholders see the results (they'll want changes/more info) then work on some grand unification design.

[worn delta]

Yeah, I'm thinking of moving in that direction. Our process is a mess and I'll need to adjust as I go, but I still need to make a presentation for grand design next week before I can even start anything. I already got "all needed information": someone tried writing logger for one type a decade ago and left it incomplete, some might or might not be obsolete.
I say devices because they are quite varied: some are just hardware boards to test wiring, some are computers with bunch of software, some are platforms to test entire functionality of final products.