Creating User Interface using AI

[autumn crest]

Hello everyone
I seen many new tools regarding ai
So I want to ask this question, I am planning to create something, that will help people to generate illustrations, ui designs according to their needs
is this something possible?
Ui like the final results, like dribbble present ui's
people often time don't know what / how to design
is it something doable to build a tool like this and solve this problem with ai?
If its possible, then how can I do it?

[hollow harness]

I remember seeing a demo that built a UI using a short textual description. A quick search on the web also shows AI that create prototypes of UIs from photos of sketches.
To do it you most likely need a dataset with lots of examples from the source domain (description or photo) to the target domain (I guess XML, HTML, or maybe directly framework code)

[autumn crest]

Just to clarify, I don't want to build a website builder, I just want to make a tool that would generate UI desings, or even illustrations design according to your business needs

Where can I get data to teach the machine?

[hollow harness]

If it is to generate design proposals and not functional prototypes, I guess you could try to fine-tune a diffusion model such as Stable Diffusion on a website screenshot dataset.
Most likely you will need to collect it yourself.

[autumn crest]

https://images-ext-2.discordapp.net/external/STUNgq0aeUEryLdBWQEWkvQ6PEmsKLYE19tRFFI8Yqo/https/cdn.dribbble.com/users/917664/screenshots/14990908/dribbble_preview_4x.png?width=814&height=610

[autumn crest]

I want to generate web designs like these and many more

[maiden idol]

I am also interested in creating ai something related to images, do we need lots of money to make a ai tool?

you said we need data for machine to learn the designs?, I am new to ai, but I scared that it will cost a lot of money to create something with it

[hollow harness]

Well, I guess you could also build a website generator by hand using procedural generation techniques, so without using machine learning and so also without using a dataset.
But the amount of work needed to build a design generator would probably be much larger, depending on what you want to do. There would also probably be less variations in the designs if you were to code a generator by hand.

Machine learning needs data, and more often than not, labelled data. You need lots of data also.
For these kinds of projects, if I had to do it I would probably collect lots of website screenshots along with some meta data (the type of website, if it is a big website, if it sells stuff, what is it selling) in order to let the AI learn what design to associate with what needs.

Take a look at how Stable Diffusion was trained and what the data looked like, you'll see what I mean

For stable diffusion, it is basically pairs of images and text

[maiden idol]

so data can be something that we can provide?

[autumn crest]

@hollow harness What about creating illustrations like these?

[hollow harness]

in general for machine learning the data is the thing you want the machine to learn. If you want to generate design, you need lots of design examples

[hollow harness]

I would do it like I explained above, I would collect lots of designs along with their meta data, and train or fine-tune a model that can generate images on the designs along with the metadata so that it can generate one from the other. I would probably fine-tune stable diffusion directly

[maiden idol]

If we put in time or work, can we teach / make an ai tool for free?

[autumn crest]

Thank you so much Ben for your guidance, it really helps me 🙂

[hollow harness]

be careful of copyrights though, I am not sure if it is completely legal to screenshot website designs and put them into a dataset without consent

[hollow harness]

I don't understand your question

[maiden idol]

how do copy.ai get all its data from?

like it helps in copywriting

so in order to teach machine learning models they might need a lots of data

How did they do that? you said about copyright issue?

[hollow harness]

copy.ai seems like a text generator, they are either connecting to an API (maybe GPT-3) or maybe they have trained their own language model on text

[maiden idol]

I am too confused about getting the data

where can I get the data

[hollow harness]

well, if they own the training data there is no copyright issue

either collect it/make it yourself
or you can outsource the work to mechanical turk for instance

getting the data is usually the difficult part, big companies can usually do it very simply because their users generate the data for them
when you are a startup, things are more complicated

[autumn crest]

As you talked about copyrighing, how would you do it in a safe / legal way and make the ai illustrations / web ui tool?

[hollow harness]

but if you have the resources, it is possible to collect it yourself, OpenAI collected millions of web pages on the web to train their language models

[hollow harness]

I would pay for the right to collect the design data or pay people to make designs that I would own and use however I want
Or maybe I would look for free designs with licenses that let me do what I want with them

[maiden idol]

if we use screenshots from dribbble or anything in order to teach the ai, how would anyone gonna know we use theirs?

[hollow harness]

As I explained, I don't know if it is legal or not to do that.
If someone can see your dataset that person could know. If they can access your AI model, the model might generate things similar to what was in the training dataset.

[maiden idol]

Thank you Ben, I am learning so much from you

[autumn crest]

Ben I want to ask your opinion, would it useful for peoples if we create a tool for web ui / illustrations with AI? Am I being too optimistic about ai generating designs? Is it something that you yourself would like to see as the AI emerges? And last but not the least, does AI have the capability to designs just like a human being?

[hollow harness]

I think the recent progress in image generation shows the AI are getting very good in creating artworks and human-like content. Check out stable diffusion, midjourney, dall-e...
Big tech companies are currently actively creating these tools, and they won't stop until they are able to automate anything. If it is not you, they will make it.

[autumn crest]

Do you consider that its an actually helpful concept to create these designs for peoples?

Because there could also be templates?

[maiden idol]

If I want to make my own ai tool, I have to train a machine learning model with a set of data and write code. I have heard that running these machine learning models on the server costs a lots of money?

is it true?

[hollow harness]

it is probably useful if you want to come up with precise designs that are not easily captured by templates, and AIs usually do create more interesting/creative variations than templates imo

[hollow harness]

well, it depends on the size of the model and dataset, training stable diffusion costs something like hundred of thousands of dollars in gpu-hours on servers, but some companies train their model on their own hardware I imagine
I think it is more or less an upper bound on the cost since you probably won't get as much data as they got for stable diffusion

[maiden idol]

yeah that was my question about making it for a less amount of money

I don't have thousands of dollars

does that mean its not for me? can I not be able to make a ai related tool?

[hollow harness]

I guess the question is more about knowing whether an AI trained at your scale would be useful

[autumn crest]

in order to make an effective ai design tool, can I start with some less data and scale further if the concept works? I know it might require a lot of data, but if done correctly can it be made useful in the initial version with less data capabilites?

effective in terms of visual designs, illustrations design etc.

with proper according to one's need

[hollow harness]

yes you can try at small scale to see if there is a chance it might work
but even before that, try to see if there are scientific papers about AI that do similar things, you can search it up on google scholar for instance

they usually explain what dataset they use, the model size and the results they get

[autumn crest]

Sure, I am doing my research, you are amazing ben, you cleared most of my doubts, I am glad that I joined this community

🙂

[maiden idol]

I also try searching for it in order to learn more about it, but couldn't find it, can you please share some resources

[hollow harness]

there is this paper that tries to do something kinda similar https://www.sciencedirect.com/science/article/abs/pii/S0167739X20330764?casa_token=zj44ky3x9xAAAAAA:ZfbMQHGZ2xTSriWETxXdw8VcSCsuvV1dlucZl1dcRlTHiCzL7XiyD6SMpsUmwpEkjT5EW6dQ2A
they collected a dataset but I haven't try to find if they made it available to the public
Search on google scholar, try keywords, I found this by typing "website design generation algorithm"

[autumn crest]

Hello Ben, I just came across this website which claims that it can create logos with the help of ai https://ainter.me/

Do you have something to share about it? How did they did this? and if they can do ai for logos than that means ai for designs / illustrations can also be made?

[hollow harness]

it is probably using a diffusion model trained on logo databases
in principle diffusion models can generate anything yeah, you just need training data and lots of computational resources

[autumn crest]

lots of computational resources? can you please tell a bit more about it

[hollow harness]

deep learning models such as diffusion models are usually trained for lots of GPU-hours on big datasets

[autumn crest]

but I don't have lots of GPU power?

[hollow harness]

then you need money to pay for training on big gpu servers, or scale down your ambitions

[autumn crest]

so are they also paying for big gpu servers too?

What if I start small with just illustrations

[hollow harness]

yeah, for training stable diffusion it costs something like 600k$

but it is trained for very long on a very large image database

[autumn crest]

I don't have thousands of dollars, but I am not really getting the point how these indie developers being able to make tools like copy.ai, jenni.ai, ai with logo etc? I am sure they didn't have invested thousands of dollars in order to train their models?
And because I don't have money, should I quit thinking about ai or to solve this problem? What would be your advice for me?

[hollow harness]

maybe they are cutting the costs by re-training existing models with less data
600k$ is kind of an upper bound in this case, most models are trained at a smaller scale, perhaps one or two orders of magnitude smaller

[autumn crest]

Hey Ben, sorry for asking this question again, I talked to few creator of these AI tools and they said that they used API for creating them. If we took this example of a AI logo generator, we would find out that its very different than stable dissuion, stable journey etc. these AI create art, but the tool above create logo which I guess is entirly different than ART.

[hollow harness]

what matters is not that they use APIs, it is what types of AIs are behind these APIs. And if you want to use the same APIs, well see if you can get access to them

[autumn crest]

in order to learn the existing stable dissuion models to be able to create logos, did they dploy their own dataset on top of stable doffisuion or there is already a preexisiting ai that created logos just like Dall E created ART?

[hollow harness]

I don't know what they did, you would have to ask them. I don't know if models specialised in logo creation exist either.

[cold seal]

yes. these're generated by stable diffusion

[autumn crest]

Do you tell stable diffusion your requirements and based on that it generates them?