🔒 PyAutoGUI users, I need your help.

[hazy crane]

I am currently working on rebuilding pyautogui that allows for adaptive resolution scaling, screen specific automation and much more.
I will get to the question but the context to understand it better:

You are a developer, you created a bot using pyautogui, and you have two screen both are at 1440p. you friend wants to use the bot you made but has one screen and 1080p screen, but you will face two issues:

1- pyautogui uses global coordinates, so all the coordinates you use will be off.
2- the images you took and used for locateonscreen, were based on 1440p screen, that means it wont work on his 1080p screen.

My solution was to create something like this:

chat_bot = mymodule.Session(screen=1, original_res=(2560,1440))

game_start = game_bot.locateonscreen("images/game_start.png",  grayscale=True, region=(85, 110, 167, 78)```

by providing the original_resolution and screen, the program will be able to compare the current users screen, with the original resolution the bot was made with, and will convert the coordinates accordingly.

I need your feedback, do you think this method is the best way to solve this problem (for both the developer and users), how would you recommend the module should handle new users (who don't know how to code) on how they will route their screens vsthe developer setup (for example them saying my game is actually on screen 1 not 0, and so on).

Please note, the module will have a GUI that does the coding heavy lifting I have attached a gif of how it works as an example (the flicking is due to the overlay being visible and stopping the locate on screen from finding the image)

[left trout]

@hazy crane

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[sullen perch]

interesting idea. though in regards to the browser; why not use browser specific tools?

this would handle all the edge cases; a window being minimized, elements out of reach, not loaded, or being at a different size/location.

you can grab any element/s via their id, class, xpath, etc.

have a look into selenium or playwright libraries.

although, I can see this being useful for interacting with UI of closed source applications or embedded frames. which accessing and inspecting elements would be slower than just configuring a plain UI flow.

[hazy crane]

Yes the python reddit page (browser) was used is just for showcase, playright and selenium are for browsers.

[sullen perch]

btw, this looks like an AI prompt 👀
just saying...

[hazy crane]

What is?

[sullen perch]

either way,

Q: do you think this method is the best way to solve this problem (for both the developer and users)
A: this is not the best way (to solve such issue)

Q: how would you recommend the module should handle new users (who don't know how to code) ...
A: Just like any other project/code, have a readme or a tutorial like guide, for new users.

also, do try and explain some requirements and method of installation in your project/guide.

[hazy crane]

If its not the best way, what do you suggest?

[sullen perch]

^@hazy crane

[hazy crane]

I already wrote that the browser was only used as showcase and is not the real use case

We are talking about pyautogui, and how to resolve its global coordinates issue

Regarding the users reading tutorials, that isnt too great either.

My vision is, you develope a bot, for a game or whatever. Your friend, who isnt IT and doesnt know how to code wants to use your bot. The goal is implement a system that makes this transaction seamless

[sullen perch]

given u have the original coords, and the original screen size - would it not be easier to just translate them to the new screen?

what do the images has to do with it?

[hazy crane]

Because locateonscreen takes images as input. If the image was captured in a 1440p then you try to locate it in a 1080p it will fail.

You need to have the original resolution it was captured in to scale up or down the image

[sullen perch]

if you want to generate the coords each time for every user- then, they'd have to go through the process which you have showen before.

marking/capturing their own pictures associated with their own unique issue/window.

on the contrary, if you're planning on running this for the same program/window on all computers; wouldnt the actions be the same?

if so, then you may just pass the coords to them, no need for the pictures in the first place. only having to translate those coords according to their new screen size.

[hazy crane]

Have you used pyautogui before?

Because it seems we are talking about different things

Using locateonscreen (what I am talking about) and using (click(x,y)) are two different things for different use cases

The point of using locateonscreen along with an image is because the image you are looking for isnt at the same place all the time

Even for click(x,y) you will need to define the resolution used to create the code to know if it needs to scale down or up for the user

For example, you have a 1440p screen, you code click(500,500). You give me the code, but i have 1080p, the click(500,500) will result in a different place to be clicked than yours

That is why defining the original_resolution is required so the program knows to scale down the click(500,500) to a 1080p screen from 1440p

[sullen perch]

3 issues I can see with this
- image scaling
- color matching (maybe use grayscale)
- if you insist on resolution being an issue, then have a set for each image, for each resolution (1k, 2k, 4k?)

imo, for static macros or workflow automation pyautogui is easy. though it kinda falls off on cross platform stuff.

I think multiple image resolutions is the only solution, if the image scaling does not work...

then again, iirc, there should be libraries that could modify img resolution.

you could make a small scripts to make every resolution version for all pictures inside a directory (recursively).

OR have it modify them during runtime to match the screen. though downgrading the resolution would be better than enhancing it. (though file size would be bigger if u use 4k)
OR have a separate version/directory for each viable resolution that the user would download/import.

[hazy crane]

Yes image scaling is the solution, but the question I was asking originally. Is how would you as a developer and user of pyautogui. Would prefer this system is made.

Would you rather input your resolutions once in the code and the system will then do the conversion based on that or do you have other suggestions

And if you created a bot and gave it to your friend, but he wants to run it on his other screen, how would like this done in the simplest way

[hazy crane]

Saving each resolution image surely does have its advantage of not requiring to ask for the original resolution. but it has many disadvantages:

1- Wasted space (creating many images of different sizes)
2- Very rigid (what if the user is using 1600x900, the code can't generate image for every possible resolution combination)

[sullen perch]

for technophobe people, I'd argue to just plug and play. have the conversion happen at runtime.

for average person, maybe have an installer? having them select which resolution they want to download and do that; or just detect it automatically.

for a bit more technical people, you can try to save some effort; have them download directly which version (or "resolution" directory) they want.

[hazy crane]

I've been thinking of having the bot run a one time setup prompt, asking the user in which screen is he planning to run this, then it will save his settings in a file.

[sullen perch]

I'd say, if you're already going with that, resize the files. it'd reduce some overhead for future runs.

like from– reading config, resize each file on load, only then try to match with screen.

to just– load img, match with screen.

their screen resolution ain't going away anytime soon. and if it does, they simply re-run the "installer"/startup script again.

[left trout]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.