[tech]: help building a .txt parser

[young pond]

The idea is to be able to parse 200+ .txt files. The size of each file could be as high as 500mb. The intent of having a parser is to parse these logs and flag all the unique errors in them by relevant keywords such as 'failure' etc. Python is preferred but other language inputs are also welcome.

@viral narwhal hi : )

[viral narwhal]

.

[viral narwhal]

doable 👍

python is best for this usecase

[young pond]

Perfect! Pls drop some ideas if you got some time :))

[viral narwhal]

im commuting rn

ill see after 8

[young pond]

No worries, ty!!

[viral narwhal]

relevant keywords such as 'failure' etc

how many such keywords?

i think this could be as simple as a 100-200 line python script

[young pond]

2-3?

[viral narwhal]

oh okay

fairly easy now

where are the err logs are stored?

[young pond]

At Max 10 tbh to have a cleaner result

[viral narwhal]

on a server right

[young pond]

On shared network

[viral narwhal]

oh

okay

[young pond]

Yess

One issue

[viral narwhal]

cron job -> trigger the script to process the new log -> this gets triggered when there is a new log file

[young pond]

The error log is inside a folder and that folder has a lot of other test related files too

[viral narwhal]

if you're going the script way, the err logs have to be in the same folder as script

[young pond]

Ahh

[viral narwhal]

if they share a common file naming, then it'd be easy

[young pond]

Then I need to manually pick them form the folders no

[young pond]

They do

[viral narwhal]

like errlog<smth-smth>.txt

so you can parse via file names

[young pond]

Yess its like that

Ahhh

[viral narwhal]

so

a shell script will be always up and running to check if a new err log (with a predefind regex pattern) file name appears in the shared network folder

[young pond]

It doesn't even have to be real time. But if it is, that is added benefit

[viral narwhal]

or you can trigger this shell script on defined times like once a day with a cron job

[viral narwhal]

i get it

so

• the error log will be stored on either a remote server/shared network folder
• a parsing microservice that will be checking for new logs -> all the unique errors will go to a database
• these will be fetch to an error monitoring dashboard like sentry etc

the parsing microservice will be fetching logs from the shared network periodically, a cron job will be very helpful

the logs will be stored with severity levels in the DB along with timestamps

the monitoring service (either a third-party software or an in-house solution) will be querying the DB to fetch data

[young pond]

I'm sorry what's a cron job?😢

[viral narwhal]

are you familiar with linux ?

[young pond]

Noo, I've been working on windows only

[viral narwhal]

oh okay

[young pond]

I can't do this on Linux either 😔 has to be done on my work laptop

[viral narwhal]

cron is basically a job scheduler in linux

[viral narwhal]

wtf

[young pond]

Okayy

[viral narwhal]

really

[young pond]

Yes i can't use my personal pc for demo

[viral narwhal]

bro we talking about microservices and damn work laptop

🤣

[young pond]

😭😭😭

[viral narwhal]

this is really smol company behavior

[young pond]

Okay i should have mentioned that before

Unfortunately they ain't small. This team mismanaged as hell

[viral narwhal]

yeah sure

[viral narwhal]

you should never tbh

[viral narwhal]

I dont use windows, dont know how to do this

[young pond]

No problem. Thanks for the ideas. I'll see if i find sth on git. Boson shared a link

[viral narwhal]

you need a demo or want a full up service running in production?

[young pond]

Demo and then full up service

[viral narwhal]

oh okay

the system design above i wrote will work

[viral narwhal]

windows in work laptop right?

do you have admin perms in it?

you can use wsl (windows subsystem for linux) to get a linux like command line

[young pond]

Yes

[young pond]

Yes i believe

[young pond]

Ahhh alr

[viral narwhal]

this fairly easy if you believe

[young pond]

😭

[viral narwhal]

a db, a parsing service, a dashboard to view errors

thats it

intern stuff

[young pond]

I've run stuff in powershell in admin mode

[viral narwhal]

ha will work

koi dikkat nhi hai

bro just write a python parser now

[young pond]

Achha okay ghar jake I'll see

[viral narwhal]

yes sure

i can do that for you but i charge hourly

sorry not sorry

[young pond]

No worries 😂I'll figure that out

[viral narwhal]

ill help nw

since the err logs do share a common file naming convention, a regular expression will help in identifying the desired files in folder containing other file types

python has inbuilt funtions to open files for parsing (combined with the regexp)

now regexp can also be used to flag the keywords "failure" etc since the file is just plain text

sets has helpful to store unique itmes

this will be the payload to put in the DB

and yeah that it

[young pond]

Can I pick a parser from git? Will that be an issue?

[viral narwhal]

wdym pick from git?

do you mean github?

oh acha

[young pond]

Haan github

[viral narwhal]

maybe yes

kisko pta chalega tbh

waise bhi chatgpt is always there

[young pond]

Haan wo BT nai hai lead se puch liya maine

He said bas implement kro

[viral narwhal]

ha toh kardo

[young pond]

Ohh yesss

[viral narwhal]

bro i dont have workign experience

mujhe nhi pata ye corporate me kya hota

[young pond]

Haan haan

Not needed to know here

[viral narwhal]

but waise you will have to make changes in it tho

as you have to put flagged logs in DB as well

coding part is easy lol, the system design is debatable

😂

[young pond]

Lemme get the code first fir dekhti hu aage ka

Bt ty so much for your help ya

[viral narwhal]

please discuss the above system design with any other coworker or lead smth

youll be appreciated

throw the mumble jumble as well

[young pond]

🥹🥹

[viral narwhal]

realtime data pipeline 🔥

[young pond]

Amen. Tyyy so much honn

If it flies i might participate with this idea in the hackathon

[viral narwhal]

containerisation and shit 🔥

[young pond]

Omg ur more corporate than me xD

[viral narwhal]

throw up some kubernetes and ansible 🔥

[viral narwhal]

🤑

need da dolla

[viral narwhal]

@slow stone

👆

[slow stone]

Don't include me in nerd shit, I logged off for today

[viral narwhal]

😔

pardon m'lady

[slow stone]

@young pond wow you nerd squad too

[viral narwhal]

senõr enzineer

[young pond]

Ayo bullying me?

[young pond]

Hi hi yes needed some help

[viral narwhal]

I think a real time data pipeline to the parsing microservice will be great, as the logging seems like high throughput

[viral narwhal]

no lol

[young pond]

I'm still in a meeting brain is going zzz

[young pond]

Yeah it's crazy. The logging

[viral narwhal]

man you're really overworked

take it slow

[slow stone]

Laugh at this loser lmao (I have a meeting at 11)

[young pond]

Bro these US guys

[viral narwhal]

damn rainforest rizz

[young pond]

AHM hai

[slow stone]

Arra

Join and chill

[viral narwhal]

[young pond]

I can't join a meeting and not pay attention

[slow stone]

It's ahm lol

Nobody pays attention lmfao

[viral narwhal]

it's not fir employees lol

[young pond]

Yeah I need to know the product roadmap

Baaki it's shit

[slow stone]

Sigh

[young pond]

I need to know if I wanna stay here next year or not 😼

[viral narwhal]

ooof

[young pond]

And market stats tell us bonus ka kya scene hoga

[viral narwhal]

please internship refferal

[slow stone]

Are you new hire or a senior engg?

[young pond]

Senior engg

[slow stone]

Ah then you'd need to lol

[young pond]

Heheh

Still junior only

[slow stone]

Damn you old tho

Wait what

[young pond]

What do u mean by senior

I'm 3 yrs exp

[slow stone]

Ohhh

[young pond]

Hbu

[slow stone]

Okay company designation might be senior engg but you mid level I assume

[young pond]

Not exactly a new hire but not a lead yet

[young pond]

Absolutely yes

[slow stone]

I meant making design docs, lead level whatever

Architect ish

[young pond]

Next year tho 😈

[slow stone]

Lesgoooo!!

All de best

[young pond]

Oh we do that . But that's cus I'm in sys arch role

New hires se bhi karwate but they don't own it completely

[viral narwhal]

I'd appreciate a sde internship refferal

[young pond]

SDE ka yaha it's not too great tbh

If you're into embedded, best hai

[viral narwhal]

still a big name in resume

I'm already getting discriminated due to degree

[young pond]

That's true

[viral narwhal]

will prove as an unfair advantage

[young pond]

Placements ka scene kaisa hai?

[viral narwhal]

state uni, no placements

only degree

[young pond]

Oh understand. With BCA it's tough too cos of the bias. I'd say keep doing dapper projects

[viral narwhal]

I'm already doing whatever i can

but still shit mentality of people

[young pond]

Efforts never go unrecognised 💯💯

I'll put in a referral if we do an off campus internship hiring next year

[young pond]

Which city if you don't mind?

[viral narwhal]

I'll be working on a real-time collaborative code editor with sandboxed environment s

[young pond]

Keep in touch with ppl in blr

Startup internships you can Target and convert in full-time

[viral narwhal]

ya ya trying

[young pond]

Get exp for 2 years, tab tak uni ka tag ka value nai eehta

[viral narwhal]

startup -> 1-1.5yoe -> big PBC

this is the plan

[young pond]

PERFECT

[viral narwhal]

as big corps generally don't hire non btech folks

[zealous citrus]

wow

basil on that grind

[young pond]

Yea I was gona say. I think btech is the min qual for tech roles :((

[young pond]

[viral narwhal]

sad

[slow stone]

Arra you work on embedded stuff that's awesome

I was looking for embedded roles during my switch, gave into money tho lol

At least I am working on C++, not embedded stuff tho but still. Loving it lol

[young pond]

Yaya

[young pond]

Haha, that's cool. Tech pay is deffo better.. this is decent too tho

[proper quiver]

I'm sure there are cronjob libraries for python. You could also just use a watchdog instead.

I also don't get why they're storing the log first and then parsing it. Corpo stuff ig :>

[elder thorn]

@young pond if this is not a pet project and something that's gonna be used in production time and time again, you can use ELK stack

Elasticsearch will search and index your logs

Logstash will push the logs to the elasticsearch

and Kibana is used for visualization and queries

[viral narwhal]

this

[viral narwhal]

dunno, do you work in corporate btw?

[proper quiver]

no

[viral narwhal]

oh okay

same

gaara's approach reflects a seasoned engineer who has worked with production environments

[elder thorn]

Yeah I have worked to set up these things before

[viral narwhal]

hey if you dont mind, what is your yoe?

[elder thorn]

5 and half

[viral narwhal]

damn, senior 🫡

[elder thorn]

Hahaha

🥹

[viral narwhal]

as a noob person, i like to duct tape code with hope

🤣

[elder thorn]

Everything is duct taped in the end

Very few products are mature

[viral narwhal]

do you have experience with microservices?

[elder thorn]

Yeah

I have worked as a Data Scientist, Software Dev and SRE

😭

[viral narwhal]

quite wide expertise

[elder thorn]

Wish I didn't

Sticking to one thing is better for mental peace

[viral narwhal]

hmm

[proper quiver]

yeah i can sense it

[viral narwhal]

hey doog are you a student?

[proper quiver]

nope i took a drop after 12th

chumma x)

[viral narwhal]

we can be friends :)

accept my req

[proper quiver]

yay

[elder thorn]

wholesome

friends 🥹

[viral narwhal]

sent one to you too 🥂

[proper quiver]

you're also my friend now

[elder thorn]

wow

accepted bbgs

[young pond]

It might be used in production but for now we're just looking for POC i guess. How does this work gaara 😭

[elder thorn]

Look it up, elasticsearch is basically a NOSQL database and you can create a cluster so that it's always up even when one or two nodes go down. Elasticsearch is specially useful for text based lookups. Logstash is like a client it can be present on multiple hosts pushing logs after parsing to the ES cluster according to what you have configured and Kibana can be used to make graphs like how many errors in past 3 hours

But you know the problem best so it might not suit your situation

You have to decide the tradeoffs

[young pond]

Oww ok ok. I'll read up on this. Ty gaara 🥹

[elder thorn]

You don't need to read up, I can help you

To decide whether it's the right fit

I'll go through this thread later to understand the scope of problem

[zealous citrus]

wow, you're hot

[young pond]

Oml i can help summarise, you'll go through everything? 😭 why are u so nice

[spiral canyon]

Error monitoring is usually done in the big companies using the ELK stack but there’s a tool called fail2ban in linux, I run a private server and I use fail2ban to monitor “errors” in the log files by defining what is an error and what action to take. If your use case is small and restricted to one system then you could use fail2ban else you’ll probably need a cluster and ELK stack

[elder thorn]

You hottest 🥹

[elder thorn]

Sure I needed an excuse to talk to you bbg

[young pond]

You shall never need an excuse to talk to me bbg

[young pond]

Thanks Ani. Sadly I can't use linux :((

[elder thorn]

That's reassuring bbg thanks for the love and support

[delicate cairn]

breh why didnt anyone invite me this looks tasty af

@young pond

[delicate cairn]

yum

[young pond]

Awwwh stahpp you

[young pond]

Ayyy arey tu server me nai tha shayad, plus serene wanted a thread to discuss so

[young pond]

Feel free to put in ur inputs

[delicate cairn]

could you send an excerpt from the logs. they should be structured ig

[young pond]

I sadly can't. It's work data

[delicate cairn]

oops

[sturdy crypt]

why you giving offis work here

[young pond]

I needed help

[spiral canyon]

Ah okay.

[viral narwhal]

any updates on this one?

[young pond]

Was de prioritised. Starting work again on it today. Need to write a rough flow/approach

New challenge is that the logs could be on various networks

[viral narwhal]

oh okay

cool

[viral narwhal]

You can deploy agents on each network to monitor new logs and push them to a common location

configure something to process the incoming logs and extract the relevant info and push it to elasticsearch

and kibana to visualisation

gaara's approach

you might as well use encryption for log data while pushing to the common service

this setup seems highly scalable and fault tolerant

the microservice approach makes the setup easy and highly customisable to specific business needs and it'll struggle in scaling

while the ELK stack is highly scalable ans tolerant

[coral geyser]

regex