#🔒 Speeding up the code

65 messages · Page 1 of 1 (latest)

finite cairn
#

Hi there, how can I make this code faster? Currently it takes one minute to generate the data.

def generate_random_tag() -> str:
    return ''.join(chr(random.randint(0, 255)) for _ in range(random.randint(1, 16)))

def generate_random_network_address() -> str:
    return str(IPNetwork(
        f"{'.'.join(str(random.randint(0, 255)) for _ in range(4))}/{random.randint(0, 32)}").cidr)

data = [
    {
        "tag": generate_random_tag(),
        "ip_network": generate_random_network_address()
    }
    for _ in range(4_000_000)
]
west fieldBOT
#

@finite cairn

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
astral fable
#

You could use random bytes

finite cairn
astral fable
#

Will be slower

finite cairn
#

What interesting is, using generator (yield):

def generate_data():
    for _ in range(4_000_000):
        yield {
            "tag": generate_random_tag(),
            "ip_network": generate_random_network_address()
        }

takes more time than using:

data = [
    {
        "tag": generate_random_tag(),
        "ip_network": generate_random_network_address()
    }
    for _ in range(4_000_000)
]
astral fable
#

List comprehension is faster it's a loop in c

finite cairn
#

Now, this code takes 45 seconds, which is 25% faster, however it is still very slow.

def generate_random_tag() -> str:
    return ''.join(chr(byte) for byte in randbytes(randbytes(1)[0] % 16 + 1 ))

def generate_random_network_address() -> str:
    ip_bytes = randbytes(4)
    ip = '.'.join(str(byte) for byte in ip_bytes)
    prefix = randbytes(1)[0] % 33
    return str(IPNetwork(f"{ip}/{prefix}").cidr)

data = [
    {
        "tag": generate_random_tag(),
        "ip_network": generate_random_network_address()
    }
    for _ in range(4_000_000)
]
astral fable
#

You can just ask for how many bytes you need and pass it directly to network

#

Then it doesn't need to parse kt

tardy helm
#

Ther problem is that you are creating a giant list called data that contains 4 million small dictionaries. (Why the "magic" number 4 million?) Perhaps it would be better to write a generator function that creates and yields the required two-entry dictionaries one at a time. You can then use such a data object by requesting one when required from the generator.

finite cairn
finite cairn
tardy helm
#

If you are performing timing test on this you may find that a lot of the time is used in creating the large list and the dicts within it. You are not only timing your two functions.

#

Try timing this instead:

for _ in range(4_000_000):
    generate_random_tag()
    generate_random_network_address()
solid knoll
finite cairn
tardy helm
solid knoll
#

do you really need the ip address in its string form when stored or just when it should be displayed?

finite cairn
finite cairn
astral fable
solid knoll
#

then yiy can do that as a last operation
but storing it as 4 bytes is more memory efficient, then you could have ot converted to string format only when needed

finite cairn
astral fable
#

IPv4Network and IPv6Network both support bytes

#

You can store it as an int in json then

solid knoll
#

but convert it to the string format when creating the json instead

finite cairn
astral fable
#

You don't need those it will load from bytes directly

solid knoll
finite cairn
astral fable
#

But it looks like you only support ipv4

finite cairn
#

ipv4 is enough for me

#

I dont need ipv6

astral fable
#

Yeah so use ipaddress.IPv4Network instead

finite cairn
#

Guys, under 19:

def generate_random_tag() -> str:
    """
    Tag of length 1..16, with full ASCII range (dec 0..255).
    """
    return ''.join(chr(byte) for byte in randbytes(randbytes(1)[0] % 16 + 1 ))

def generate_random_network_address() -> str:
    ip_int = getrandbits(32)
    prefix = randbytes(1)[0] % 33
    network_int = ip_int & (0xffffffff << (32 - prefix)) & 0xffffffff
    network_address_cidr = str(IPv4Address(network_int))
    return f"{network_address_cidr}/{prefix}"


data = [
    {
        "tag": generate_random_tag(),
        "ip_network": generate_random_network_address()
    }
    for _ in range(4_000_000)
]
#

is it possible to make it in like 1 sec or so?

solid knoll
#

try skipping the use of IPv4Address all together

finite cairn
#

so like do not use this library at all?

solid knoll
#

maybe try something like

from socket import inet_ntoa

def generate_random_network_address() -> str:
    ...
    return inet_ntoa(network_int) + "/" + prefix
finite cairn
#

If this code is correct ofc I might have written something wrong here

solid knoll
#

i think you should be able to use the inet_ntoa() function from the socket library instead of doing that part manually, might even be more optimized since i believe it's implemented directly in C

finite cairn
solid knoll
#

very small performance win compared to using the library, i thought it would make a bigger difference 😞

finite cairn
#

I suppose this will be hard to beat:

def generate_random_tag() -> str:
    """
    Tag of length 1..16, with full ASCII range (dec 0..255).
    """
    return ''.join(chr(byte) for byte in randbytes(randbytes(1)[0] % 16 + 1 ))


def generate_random_network_address() -> str:
    ip_int = getrandbits(32)
    prefix = randbytes(1)[0] % 33
    mask = 0xffffffff << (32 - prefix)

    network_int = ip_int & mask & 0xffffffff

    return f"{inet_ntoa(network_int.to_bytes(4, byteorder='big'))}/{prefix}"


data = [
    {
        "tag": generate_random_tag(),
        "ip_network": generate_random_network_address()
    }
    for _ in range(4_000_000)
]
solid knoll
#

compared to your starting point of taking about 60 seconds it's at least a pretty substantial improvement

solid knoll
finite cairn
#

13,3

# 
def generate_random_tag() -> str:
    """
    Tag of length 1..16, with full ASCII range (dec 0..255).
    """
    length = randbytes(1)[0] % 16 + 1
    return randbytes(length).decode("latin-1")
#

like this?

#

latin-1 decodes ok for 0-255 ascii

solid knoll
#

yeah, something like that, but i just made it into a single line

finite cairn
#

I thought of making this network address directly in bytes but idk it seems to be even slower than doin this using ints

solid knoll
#

that's all i can think of right now other then inlining the generate_random_tag function as it's only a single line of code anyways, like:

data = [
    {
        "tag": randbytes(randbytes(1)[0] % 16 + 1).decode('latin-1'),
        "ip_network": generate_random_network_address()
    }
    for _ in range(4_000_000)
]
``` but i don't expect that to be a very big difference at all
finite cairn
#

yeah this does not affect nearly at all the speed

solid knoll
#

nah, it shouldn't, that is really a micro optimization which really hurts readability at the same time

finite cairn
#

Thank you so much for your effort, if you have any further ideas you can pm me I will appreciate the help 🙂

solid knoll
finite cairn
#

Thank you.

#

!close

west fieldBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.