🔒 ```

[boreal helm]

python
# Use regex to find lines containing "<Error>" in stdout_str
pattern = r'.*<[^<]Error.>'
error_lines_stdout = re.findall(pattern, stdout_str,re.MULTILINE)
print (str(error_lines_stdout))

return JSONResponse(content={"error_lines_stdout": error_lines_stdout})

i want from the error part and the rest of the line```

[urban forge]

@boreal helm

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[boreal helm]

@toxic cairn

[toxic cairn]

here we are

one moment, a work thing came up

[boreal helm]

oh great 🙂

[toxic cairn]

alright im back

so regex is its own language, and it is very complex

but you can solve most problems by knowing a few things

so ill be talking about those few things instead of doing a more clever and complicated solution

just know that you can do alot

[boreal helm]

ah

[toxic cairn]

regex sees the characters

one character at the time

[boreal helm]

i have file which has 10000 line so the command validate each line

so thats why i am going for regex

[toxic cairn]

so when you have a string that looks like

[boreal helm]

what do you think the best way to do that because i also have file with 40000 lines

[toxic cairn]

you will have a lot of extra junk

[boreal helm]

yes

[toxic cairn]

so lets focus on what we talked about earlier, as i think that is the important part

[boreal helm]

yes

[toxic cairn]

we need to look for <Error> but we have to add more checks along the way

[boreal helm]

yes also

i need the rest of line after <Error>

[toxic cairn]

yes, thats the final part in this

[boreal helm]

ya

[toxic cairn]

the first character we need to look for is <

since regex searches for characters, we can just use this verbatim as it is

the pattern starts out with <

so what is the next character we need to look for

[boreal helm]

alright

[toxic cairn]

well that is whaterver character that follows, as these are the color characters

[boreal helm]

ya

[toxic cairn]

so we dont know exeactly what that character is, we use the single character wildcard .

and we want to look for it any number of times

* matches 0 to many of the previous thing

[boreal helm]

alright

[toxic cairn]

+ matches 1 to many of the previous thing

? matches 0 or 1 of the previous thing

[boreal helm]

oh

[toxic cairn]

so here we can even say, that < needs to be mached 1 or many, or 0 or 1, but leaving it just as < means it must be there

<+ also makes sense here

[boreal helm]

<.*>?

[toxic cairn]

so to match the colour junk, we need to add .* to our pattern

<.*

[boreal helm]

alright

[toxic cairn]

so far so good right?

[boreal helm]

ya

[toxic cairn]

everything clear so far?

now we look for verbatim text

[boreal helm]

ya

[toxic cairn]

Error

[boreal helm]

yes

[toxic cairn]

now, maybe you want to look fro both error or Error

but for now we can just use Error

[boreal helm]

as it output it always <Error>

[toxic cairn]

<.*Error

[boreal helm]

ye

[toxic cairn]

you can group things together, so instead of searching for 1 or 0 of the previous thing (a character) you can use it to serach for a group of things

(error|Error) as an example

[boreal helm]

yes

[toxic cairn]

or (e|E)rror

[boreal helm]

yes

[toxic cairn]

but lets keep it Error for now, just know that even though i say, regex searches one char at the time

it also has the ability to search for whatever you make a thing to be

[boreal helm]

yes

<.*Error is good right now

[toxic cairn]

so after Error, we need a number of colour junk

[boreal helm]

yes

[toxic cairn]

the same as before, . to match any wildcard char and * to get any number of them

or zero

<.*Error.*

[boreal helm]

<.*Error.*?

[toxic cairn]

yes

and now we can add the > part as well

[boreal helm]

<.*Error.*>

[toxic cairn]

yes, this rule will look strange alone, as we tell it to look for any number of characters and then end at >

so we dont really know when it will end, there can be multiple > in that line

[boreal helm]

yes

[toxic cairn]

but since we already want to continue the full length of the line, we dont have to worry about it

[boreal helm]

yes

[toxic cairn]

we could say that it should stop at the first match of >, but for this problem we dont need that

most regexes can be written pretty simple

[boreal helm]

yes we need till end of line

[toxic cairn]

so after the > we need to loof for any wildcard characters, 0 to many times, until the end of the line

well, the first part of the finishing pattern we have done before, thats .*

<.*Error.*>.*

[boreal helm]

yes <.*Error.*>.*

[toxic cairn]

there is something called anchors in regex

those are characters that are fixed points in the syntax

[boreal helm]

got it

[toxic cairn]

^ means matches the beginning of the string

and $ matches the end of the string

[boreal helm]

<.*Error.*>.*^$

[toxic cairn]

meaning the full regex is <.*Error.*>.*$

notice i only used one anchor

as there are no matches for error before the line begins

[boreal helm]

oh yes

[toxic cairn]

this is our pattern, now lets get it to actually work

as it will probably not work alone

re.findall(pattern, out) you use this right?

[boreal helm]

yes

    pattern = r'<.*Error.*>.*$'
    error_lines_stdout = re.findall(pattern, stdout_str,re.MULTILINE)
    print (error_lines_stdout.decode().strip())

[toxic cairn]

so to make the regex use multiple lines, meaning it will parse \n as a newline

[urban forge]

Hey @boreal helm!

It looks like you pasted Python code without syntax highlighting.

Please use syntax highlighting to improve the legibility of your code and make it easier for us to help you.

To do this, use the following method:
```py
print('Hello, world!')
```

This will result in the following:

print('Hello, world!')```
You can **edit your original message** to correct your code block.

[boreal helm]

    pattern = r'<.*Error.*>.*$'
    error_lines_stdout = re.findall(pattern, stdout_str,re.MULTILINE)
    print (error_lines_stdout.decode().strip())

[toxic cairn]

you have to set a flag

re.findall(pattern, out, flags=re.MULTILINE)

this should be a list of length 3

[boreal helm]

print (error_lines_stdout.decode().strip()) this give me AttributeError: 'list' object has no attribute 'decode'

[toxic cairn]

findall returns a list

result = re.findall(pattern, out, flags=re.MULTILINE)

[boreal helm]

but it reurn me like this

[toxic cairn]

print(*result) shoould work just fine here

[boreal helm]

  "error_lines_stdout": [
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_DIRECTION\u001b[0m(\u001b[33m189\u001b[0m)] - \u001b[31m\"-\" is not a valid direction modifier, \"->\" and \"<>\" are supported.\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_SIGNATURE\u001b[0m(\u001b[33m39\u001b[0m)] - \u001b[31merror parsing signature \"alert ip any any - any any (msg:\"SURICATA Applayer Wrong direction first Data\"; flow:established; app-layer-event:applayer_wrong_direction_first_data; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260001; rev:1;)\" from file /var/lib/suricata/rules/test.rules at line 2\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_NO_RULES_LOADED\u001b[0m(\u001b[33m43\u001b[0m)] - \u001b[31mLoading signatures failed.\u001b[0m"
  ]```

[toxic cairn]

well, i just used the result as a list in my example above

[boreal helm]

how to get rid of this junt charactes

[toxic cairn]

didnt you need them?

[boreal helm]

no

[toxic cairn]

print them and see

[boreal helm]

"error_lines_stdout": [
"<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_DIRECTION\u001b[0m(\u001b[33m189\u001b[0m)] - \u001b[31m"-" is not a valid direction modifier, "->" and "<>" are supported.\u001b[0m\r",
"<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_SIGNATURE\u001b[0m(\u001b[33m39\u001b[0m)] - \u001b[31merror parsing signature "alert ip any any - any any (msg:"SURICATA Applayer Wrong direction first Data"; flow:established; app-layer-event:applayer_wrong_direction_first_data; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260001; rev:1;)" from file /var/lib/suricata/rules/test.rules at line 2\u001b[0m\r",
"<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_NO_RULES_LOADED\u001b[0m(\u001b[33m43\u001b[0m)] - \u001b[31mLoading signatures failed.\u001b[0m"
]

this is the print resulu

[toxic cairn]

well, you have to print the actuall string to see the output as "normal"

if you use my example from above, you can see this in action, i unpack the list while im printing

to remove the colours you to search and replace them all

[boreal helm]

it hang out my api

@app.post("/upload-file/")
async def upload_file(file: UploadFile = File(...)):
    # Save the uploaded file temporarily
    with open(file.filename, "wb") as temp_file:
        temp_file.write(await file.read())

    # Move the uploaded file to /var/lib/suricata/rules/
    destination = f"/var/lib/suricata/rules/{file.filename}"
    shutil.move(file.filename, destination)

    # Run the command on the uploaded file
    command = f"docker exec -it --user suricata suricata suricata -T -S {destination}"
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()

    # Decode stdout and stderr
    stdout_str = stdout.decode().strip()
    stderr_str = stderr.decode().strip()

    # Print subprocess output
    print("STDOUT:")
    print(stdout_str)
    print("STDERR:")
    print(stderr_str)

    # Use regex to find lines containing "<Error>" in stdout_str
    pattern = r'<.*Error.*>.*$'
    error_lines_stdout = re.findall(pattern, stdout_str,re.MULTILINE)
    print (*error_lines_stdout)

    return JSONResponse(content={"error_lines_stdout": error_lines_stdout})

[toxic cairn]

it might be better to have the command not print out colours in the first place if you dont want that

[boreal helm]

ok wait

 <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert ip any any - any any (msg:"SURICATA Applayer Wrong direction first Data"; flow:established; app-layer-event:applayer_wr <Error> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - Loading signatures failed.col-command-decode; sid:2260001; rev:1;)" from file /var/lib/suricata/rules/test.rules at line 2

i get two

oh wait

in response i get 3

{
  "error_lines_stdout": [
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_DIRECTION\u001b[0m(\u001b[33m189\u001b[0m)] - \u001b[31m\"-\" is not a valid direction modifier, \"->\" and \"<>\" are supported.\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_SIGNATURE\u001b[0m(\u001b[33m39\u001b[0m)] - \u001b[31merror parsing signature \"alert ip any any - any any (msg:\"SURICATA Applayer Wrong direction first Data\"; flow:established; app-layer-event:applayer_wrong_direction_first_data; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260001; rev:1;)\" from file /var/lib/suricata/rules/test.rules at line 2\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_NO_RULES_LOADED\u001b[0m(\u001b[33m43\u001b[0m)] - \u001b[31mLoading signatures failed.\u001b[0m"
  ]
}

on terminal i get two ?

[toxic cairn]

the list is 3 long

dont trust output, your human eyes is faulty

[boreal helm]

but it has again junk character

no no

on terminal there is two

 <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert ip any any - any any (msg:"SURICATA Applayer Wrong direction first Data"; flow:established; app-layer-event:applayer_wr <Error> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - Loading signatures failed.col-command-decode; sid:2260001; rev:1;)" from file /var/lib/suricata/rules/test.rules at line 2

[toxic cairn]

its easy to miss things when you constantly change. just make sure that the data your working with comes from the subprocess result

[boreal helm]

yes its coming from subprocess

[toxic cairn]

what thing are producing this output?

[boreal helm]

anyway

like from which command the output is generating ?

[toxic cairn]

yes

[boreal helm]

    # Run the command on the uploaded file
    command = f"docker exec -it --user suricata suricata suricata -T -S {destination}"
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()

[toxic cairn]

so its a docker exec command that does something

what is the something it does?

[boreal helm]

yes

basically i take file as input

and check rules syntax

for 2 rules as test there is two error because i remove ) from them to test

[toxic cairn]

i would remove colours from the docker instead of doing it after the fact.

but if you cant do that, you have to write a function that cleans up the string

[boreal helm]

but in response of api i get this

{
  "error_lines_stdout": [
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_DIRECTION\u001b[0m(\u001b[33m189\u001b[0m)] - \u001b[31m\"-\" is not a valid direction modifier, \"->\" and \"<>\" are supported.\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_SIGNATURE\u001b[0m(\u001b[33m39\u001b[0m)] - \u001b[31merror parsing signature \"alert ip any any - any any (msg:\"SURICATA Applayer Wrong direction first Data\"; flow:established; app-layer-event:applayer_wrong_direction_first_data; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260001; rev:1;)\" from file /var/lib/suricata/rules/test.rules at line 2\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_NO_RULES_LOADED\u001b[0m(\u001b[33m43\u001b[0m)] - \u001b[31mLoading signatures failed.\u001b[0m"
  ]
}```

this is correct i need this but need to get rid of from this junk character

[toxic cairn]

do you want to see my regex notes? i keep them on hand for whenever i write regex

[boreal helm]

no i dont need

[toxic cairn]

!d str.replace

[urban forge]

str.replace

str.replace(old, new[, count])```
Return a copy of the string with all occurrences of substring *old* replaced by *new*. If the optional argument *count* is given, only the first *count* occurrences are replaced.

[boreal helm]

@app.post("/upload-file/")
async def upload_file(file: UploadFile = File(...)):
    # Save the uploaded file temporarily
    with open(file.filename, "wb") as temp_file:
        temp_file.write(await file.read())

    # Move the uploaded file to /var/lib/suricata/rules/
    destination = f"/var/lib/suricata/rules/{file.filename}"
    shutil.move(file.filename, destination)

    # Run the command on the uploaded file
    command = f"docker exec -it --user suricata suricata suricata -T -S {destination}"
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()

    # Decode stdout and stderr
    stdout_str = stdout.decode().strip()
    stderr_str = stderr.decode().strip()

    pattern = r'<.*Error.*>.*$'
    error_lines_stdout = re.findall(pattern, stdout_str,re.MULTILINE)
    print (*error_lines_stdout)

    return JSONResponse(content={"error_lines_stdout": error_lines_stdout})

[toxic cairn]

!d str.translate

[urban forge]

str.translate

str.translate(table)```
Return a copy of the string in which each character has been mapped through the given translation table. The table must be an object that implements indexing via [`__getitem__()`](https://docs.python.org/3/reference/datamodel.html#object.__getitem__), typically a [mapping](https://docs.python.org/3/glossary.html#term-mapping) or [sequence](https://docs.python.org/3/glossary.html#term-sequence). When indexed by a Unicode ordinal (an integer), the table object can do any of the following: return a Unicode ordinal or a string, to map the character to one or more other characters; return `None`, to delete the character from the return string; or raise a [`LookupError`](https://docs.python.org/3/library/exceptions.html#LookupError) exception, to map the character to itself.

You can use [`str.maketrans()`](https://docs.python.org/3/library/stdtypes.html#str.maketrans) to create a translation map from character-to-character mappings in different formats.

See also the [`codecs`](https://docs.python.org/3/library/codecs.html#module-codecs) module for a more flexible approach to custom character mappings.

[toxic cairn]

these two can clean up your string

[boreal helm]

should i apply them on error_lines_stdout

[toxic cairn]

on each of the strings yes

[boreal helm]

str.translate(error_lines_stdout)

[toxic cairn]

not the list ofc

stick with replace if you dont know any other

[boreal helm]

ah i am confuse where should i apply that

return JSONResponse(content={"error_lines_stdout": error_lines_stdout}) on this ?

[toxic cairn]

what does re.findall return?

[boreal helm]

 <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert ip any any - any any (msg:"SURICATA Applayer Wrong direction first Data"; flow:established; app-layer-event:applayer_wr <Error> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - Loading signatures failed.col-command-decode; sid:2260001; rev:1;)" from file /var/lib/suricata/rules/test.rules at line 2

and in json respone

[toxic cairn]

no, i meant for you to answer not show me the value.

[boreal helm]

{
  "error_lines_stdout": [
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_DIRECTION\u001b[0m(\u001b[33m189\u001b[0m)] - \u001b[31m\"-\" is not a valid direction modifier, \"->\" and \"<>\" are supported.\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_INVALID_SIGNATURE\u001b[0m(\u001b[33m39\u001b[0m)] - \u001b[31merror parsing signature \"alert ip any any - any any (msg:\"SURICATA Applayer Wrong direction first Data\"; flow:established; app-layer-event:applayer_wrong_direction_first_data; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260001; rev:1;)\" from file /var/lib/suricata/rules/test.rules at line 2\u001b[0m\r",
    "<\u001b[1;31mError\u001b[0m> - [\u001b[33mERRCODE\u001b[0m: \u001b[31mSC_ERR_NO_RULES_LOADED\u001b[0m(\u001b[33m43\u001b[0m)] - \u001b[31mLoading signatures failed.\u001b[0m"
  ]
}

like it give the result of rows with string ?

like it find Error and return those lines

[toxic cairn]

yes, but in a more techincal way it returns a list of strings

agree to my def?

[boreal helm]

yes then we unpack it

ye

[toxic cairn]

and those strings, those you need to change and remove the ansi colour codes

[boreal helm]

yhes

[toxic cairn]

i would use regex to remove it, but you can use str.replace as well

[boreal helm]

regex is good

because i think for file which has 40000+ line regex is good

i did it

[toxic cairn]

do you want me to share you a function i have used in the past to do this? or are you good?

[boreal helm]

def remove_ansi_color_codes(text):
    ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
    return ansi_escape.sub('', text)```

def remove_ansi_color_codes(text):
    ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
    return ansi_escape.sub('', text)

@app.post("/upload-file2/")
async def upload_file(file: UploadFile = File(...)):
    # Save the uploaded file temporarily
    with open(file.filename, "wb") as temp_file:
        temp_file.write(await file.read())

    # Move the uploaded file to /var/lib/suricata/rules/
    destination = f"/var/lib/suricata/rules/{file.filename}"
    shutil.move(file.filename, destination)

    # Run the command on the uploaded file
    command = f"docker exec -it --user suricata suricata suricata -T -S {destination}"
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()

    # Decode stdout and stderr
    stdout_str = stdout.decode().strip()
    stderr_str = stderr.decode().strip()

    # Remove ANSI color codes from stdout_str and stderr_str
    stdout_str_clean = remove_ansi_color_codes(stdout_str)
    stderr_str_clean = remove_ansi_color_codes(stderr_str)

    # Define regex pattern to match lines containing "<Error>"
    pattern = r'<.*Error.*>.*$'

    # Use regex to find lines containing "<Error>" in stdout_str_clean
    error_lines_stdout = re.findall(pattern, stdout_str_clean, re.MULTILINE)

    # Print the cleaned error lines without ANSI color codes
    print(*error_lines_stdout)

    return JSONResponse(content={"error_lines_stdout": error_lines_stdout})

[toxic cairn]

that is exactly what i also did i think

[boreal helm]

oh great

[toxic cairn]

the ansi escape code pattern is very deterministic

[boreal helm]

thanks @toxic cairn so much for yourl help

can you just tell 1 thing

[toxic cairn]

let me just find mine and see..

tell me sure!

[boreal helm]

if i need to put if condition

like if error_lines_stdout has Error return file has syntax error if there is no Error return syntax is correct

[toxic cairn]

import re


def remove_ansi_escape_sequences(text: str) -> str:
    """
    Remove ANSI escape sequences from text.
    :param text: str of text.
    :return: str of text with ANSI escape sequences removed.
    """
    ansi = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
    return ansi.sub('', text)

[boreal helm]

how to return that

[toxic cairn]

im not sure i understand

[boreal helm]

let me try my end first

then i ping you if i cant

[toxic cairn]

the regex will search for the Error part of the string and match that

if it does not have that, it will not match it

[boreal helm]

yes thats correct is it good for file which has 40000+ lines?

ok got it i'll try if part on it

[toxic cairn]

yes, no worries with such a small file

[boreal helm]

alright @toxic cairn thank you so much for your help

[toxic cairn]

my pleasure! good luck

[boreal helm]

👍

[urban forge]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.