#Neuro rants database
1 messages · Page 1 of 1 (latest)
i don't even know where to start 
The idea seems interesting. I can at least provide hosting. Maybe I'll even learn how to build a web page, who knows.
Trying to test something like this would require some starting data though
I do think my system with 25TB of storage would be a good host system
I can probably make a simple website, depending on what's needed
Ok, I can host it then
As I've said many times, my PC is on pretty much 24/7 anyway, and mostly just sitting around. Even the specs are closer to a server than a PC, so it's a pretty good host for something like this.
I'll try make an example website sometime soon, I'll have to relearn some stuff though as it's been a while since I've coded
Well, if you need anything, let me know.
will do
What would really be nice if a certain special someone could output Neuros speech logs 😉 but I think he'll keep that private.
As for collecting via auto closed caption could be a method.
As for hosting, look into Google cloud SQL. A text based database isn't large at all.
But doesn't a host like that still cost something? I'd host this for free with probably way more customization
Who needs the cloud when I've got 25TB sitting around
And what if we want to add the video for the rant on the website itself?
I'd say my hosting capacity is better for something like this, it has good uptime and is free
And it is quite massive
The CCs idea sounds good to get started. Basically NArchiver hosts lots of Neuro streams on YT and CCs could be downloaded from there. Processed CCs could go into some full-text search engine and then later, while performing searches, the site could generate YT links with timestamps. There's one problem though, quite a few of NArchiver videos doesn't have auto generated CCs, but I suspect he could fix it if enough of us asked him.
https://www.youtube.com/@NArchiver/videos
That could work, if it was able to be implemented. I think my system wouldn't even care if it needed more compute. If you need processing power to process the CCs, I have plenty to go around
I was thinking we could add the rants manually or would that be too much work
I don't know, whatever works.
I made a 1180 file TTS dataset from Evil manually, so a couple rants would be nothing (probably)
I don't really care how you get the rants. I'll store the data all the same no matter the origin
So what kind of features do we want? I could quickly come up with a filestructure for it
Here's what I came up with from my head
Just something pretty random, but I think it would work
It would also be pretty easy to build by hand or with a program, whichever one we choose
I can modify it if needed
This is just an idea
yeah this makes sense
It has a place for raw text, a link to something like a video, and keywords we might want to have some rants searchable by, even though they don't directly contain them
And to know when it's ended, just find the tags and look in between. Even if there was some random data past them, they are a good indication of where the properly formatted data is
For reference, here are CCs in ttml format from the latest NArchiver video (ttml is probably the least cluttered format). The automatic CCs aren't perfect, but it's easy to see that even a manual search through such files could serve as a decent poor man's solution.
If you guys want to do the whole work by hand, I most likely won't be able to help much with that.
That has a stupid amount of data that would need to be stripped out
Getting data from such XMLs is easy.
I don't even know if autogenerated CCs are that accurate. Going trough Neuro's subtitles with image recognition would probably be more accurate
Well, I only suggested some relatively easy, fast and decently good solution. Perhaps others will have better ideas.
Maybe we just get a bunch of people to send in the rants and we add them to the file manually
that's what I was thinking
The simplest solution is usually the best one
Do we need any more features for our file format, or is the one I came up with good?
I wonder how many rants I have randomly in my few hundred gigabytes of Neuro recordings
Also I added a date field to the filestructure
what formats do we use for dates
Could be something like "24-1-29"
so year month day
So it would look a bit like this (just imagine those as numbers)
That would only start to become an issue once the year 2100 comes around, but that's quite the ways away still
And if this is still up by then we just add the 20/21 at the beginning and automate converting all the data
I do wonder how much data this will actually be, but I doubt It'll be more than a few gigabytes, definitely not a terabyte or anything close to that
Here's how a slightly larger database would look with these default values
The scaling with this fata format is actually pretty easy, just add more lines below the current ones and then move the end tag "</data>"
Maybe the interface of the website could be something similar to a search engine, where the rants would show up in full text form with their dates above them, then have the text be clickable to go to the included link
Similar to how things show up on a Google search
I can definitely set up everything we'll need on my side, but if we want to have this as a proper website, we'll need a domain that I can host it from, so people don't have to type in my PC's IP to get into the website.
Preferably something I can host other things from too if needed
Seems like there might be ways to get domains for free, but I'm gonna look into it more at some point
If anyone has a domain for this already, that would probably be better
Any progress on building the part that I can host?
I've been planning on adding the iconic rants into the txt file but couldn't find time to do that
Ok, that could be a good idea if we stick with this format, which I think is good
The more data we have ready to test with when the website part is done, the better
Just send me the data once you've added some and I'll make sure it matches the data format I came up with
So i backread a bit and tried to understand... but im dum 
isnt better just to make some random ass json where ppl can submit their rants? and whoever decides to make the website just use that file, idk something like this
Yeah, I think I like that one. We should build around that kind of format. We could make a feature in the website that lets people submit rants with all that data included. (with some moderation of course)
We can add all the rants into the database directly, but just add a data value only we on the serverside can see that determines if the rant is shown on the website
My format was an example anyway
Any thoughts from anyone else?
I’d say this would be a good way to do it https://discord.com/channels/574720535888396288/1192664583681347664
That would be for the interface though? The datastructure we are trying to come up with is supposed to be how the data itself is stored internally and is not directly shown to the user, being a part of the backend
I also don't know if that UI would quite work with the features for this system
I thought of a possible funny addition by the way: a button to download the whole rants database for no reason
Just not sure if that would be easy to handle for my "server"
Or mainly by the network attached to it
By the way, any prediction when we'll have something to test for the website hosting side?
yo
how is this going?
who's doing what?
I can do frontend if needed, (I can do backend too, if nodejs is okay)
@desert terrace what exactly do you mean by
provide hosting
I'd rather use JSON if you don't mind
it'll be wayy easier to work with
I don't think there are more than a couple megs of rants tbh, so I assume size isn't really an issue
it would also help immensely if someone provided an example of a couple different types of rants
progress so far
I'd appreciate some recommendations for colors for the page and stuff, I don't really have the eye for that
I mean that I can store it and have it accessible from my hardware, basically running the website there.
what environment?
I'll figure it out. If you mean OS, I run under Windows, but can technically setup a hypervisor for more OSs to run in parallel if needed.
nah it's fine
I'll be using express, which runs on node.js
the LTS version should be fine
If I can host it, it's fine by me what it uses
By the way, I'm probably gonna be hosting off of my G: drive, but would it be possible to automatically copy the data to the E: and F: drives for backup reasons?
yep. I can do that
Cool, do you need me to specify filepaths?
I won't be adding that just yet
I'll make a setting file when I do
so you can specify where you'd like the backup to go to
Oh ok sounds good.Let me know if you need anything
I'm about to push my progress, so you can download that and try running it
Where will you push to?
Size is absolutely not an issue, I have 25TB in this thing
Oh, you set up a Github repository, that's cool. Sure, I can try it
Just let me know when and how to run it
do you have git?
Seems like I downloaded it at some point
okay
after installing node.js (LTS version)
pick a folder where you want the project to run from
#folder
git clone git@github.com:NotRustyBot/neuro-rants.git
#if you don't have ssh, use this instead:
git clone https://github.com/NotRustyBot/neuro-rants.git
#folder/neuro-rants/client
npm i
npm run build
#folder/neuro-rants/server
npm i
npm start
to start the server again later, you only need to
#folder/neuro-rants/server
npm start
Just wait like 15 minutes I'm gonna go remote real quick, be back as soon as I can
sure
So how should I do the git stuff? I really don't know how to use git yet, (I can figure it out) since I really ever only used it once for something
after picking a folder where you want the project to be in
open up a terminal in that folder
and run
git clone git@github.com:NotRustyBot/neuro-rants.git
it should just work
G:\hostStuff\NeuroRantsDatabase Is this path good?
should be okay
What kind of terminal do I open?
cloning the git repo will create
neuro-rants
inside the folder, so you can maybe run it in G:\hostStuff
so the actual project will be in G:\hostStuff\neuro-rants
Ok, sounds good
on win 11 you can rightclick > open terminal
Win 10?
on win 10 you can type cmd here
in file explorer
which will open the terminal in the that directory
So just regular CMD works?
yep
if that doesn't work for you, it's possible you installed git to only be launched via the git bash
and then, depending on what were your install settings, you can maybe rigtclick and open git bash in that directory
and use that instead
What do I do with this?
type yes
That one did something
did the neuro-rants folder get created?
It did appear, and has contents in it
Done
That doesn't work in CMD
do
node -v
What does that do? I've never seen that command
node -v
should show you the node.js version
if it gives you an error, then node probably didn't install properly for some reason
have you installed node.js ?
Couldn't run this
I'll just try the bash thing
node is usually installed manually, from
https://nodejs.org/en/download/
you want the LTS version
windows 64bit
the installer will ask you if you want to install complementary software as well (such as npm), make sure that option is selected
I think it is by default, but I may be misremembering
after it's done, you will need to open a new terminal.
navigate back to the client folder
cd G:\hostStuff\neuro-rants\client
and run the install and build commands again
Now it worked
alright
there should be a few more folders in client
specifically node_modules and build
How does the syntax for that work?
sorry I'm not sure what do you mean
What do I type to specify it
oh no I just meant that if you go and check the folder, there will be new folders created
the client folder should look like this now
I noticed some of those
- build folder
ah, ok
now, navigate to the server
cd ..
cd server
and run
npm i
to download the served dependencies
and finally, start the server
npm run start
you should see something like
Server is running at http://localhost:3000
cool
next time, you only need to run the last command
Ok cool. I think I'm gonna figure out how I can set a url for that at some point and try to forward that port
Probably gonna forward the port once I'm back, since I can't remotely forward any ports
in the end it should run from port 80 (or 443) as a regular website. I'll add that to the settings as well once I get around
I'll do that tomorrow
Oh, so I should forward that instead? Or figure it out once I do the url stuff?
I think I'm gonna see about getting a domain pretty soon
So people don't have to type in my IP when they want to open the database
yea, port 3000 is used here just for testing, once it's actually ready it needs to run from 80 or 443
OK, sounds simple enough
I'll go hit the hay
need something before I go?
Don't think so. I'll figure out a domain probably today, so that'll be something for me to figure out
okay, good luck
How do I stop the server actually?
try ctrl+c?
Ok, that worked
That seems like something that would work for this, so I think we have a website builder now
Made a quick batch that can start the thing at startup once it's done
Having issues with doing the domain thing, does anyone have a domain that could be utilized?
Seems like all the free domains are not available or not usable for this purpose
So I think we need to find some domain to host on. If anyone has a suitable domain to use for this, or a way to get it to work, would be really useful right now
what was the issue with the ones you tried?
They were not available or not able to set an IP adress as a target
Do you know of any functional way to get a domain attached to an IP?
when you get a domain you get access to its DNS records
The issue with that is basically all the proper free domains are unavailable or I can't find the way to get them
And all the budget went into the PC, so I could like barely get a cheap domain if it was absolutely neccessary
eu domains are very cheap. You can also ask the harrisontemple.com ppl for a subdomain if you want
I can ask, but how should I ask?
By the way, I'm gonna be on my PC physically again soon. Should I forward any ports?
(soon in this case is around an hour at most)
You can try forwarding 3000, then we test if that connects
If so, you can close 3000 and open 80
Ok, I'll let you know once I'm ready to test
I'm on my PC now and will be beginning the test
okay
It does indeed work
great
Should I now change the forward to port80?
yes
okay, now I just need to finish the app
And we need a domain to host under
In the website itself, we should have a feature to allow users to submit rants if they have all the required data, but not immediately make them visible, but instead have a flag that is updated after the submission is verified manually
yep
That would make it easier to collect the data, but we don't want to allow absolutely everything
Any new information about the domain situation?
By the way, I scrolled up in the harrison temple thread thing and found this: #1184082970341031997 message Any idea if it could be relevant?
How's the application development going?
I added the ui for submitting rants
After I add the approval tool it should be good to go
Ok, sounds like it's going well. Any idea if we can get a domain from the harrison temple people?
I DMd Rhae, no response so far
I guess we'll wait then. Let's hope it doesn't take too long
We can probably do all the setup for the website while we wait
At least host side
Just tell me once the finished application needs installing or if you need anything
sure
Hey I got the message and I'm down for it. Depending on how it's hosted I am good to help get this on a harrisontemple domain.
As far as I can tell no one is using them for abything now so I can put the domains on your nameservers or get them to work with whatever DDNS you're using if you dont have a static IP
I do have a static IP
Let me know when we'll set it up and if you need anything
It'll be hosted off of my system, which is pretty much always on anyway and has a static IP I use sometimes.
Any idea when we can set up the domain stuff? I can do it whenever (Even remotely(excluding when I sleep))
I'm fairly sure Rhae only needs to have your IP, the rest of the setup is on Rhaes side.
DMs, but it doesn't really matter
I have never started a DM, how do you do that?
Click on Rhae, click Message
I think I found the button, I'll try to get the IP to them
Hope that worked
I guess now you just need to finish the application, unless you already did
I didn't
Maybe today, tomorrow at latest
Ok, I'll have plenty of time to install it once it's done, so once it is, just let me know and we can install it
Also, for the data backup, can you make it save the backups maybe daily, and into E:\rantsBackup and F:\rantsBackup?
Hey I kind of want to get some community feedback before I make a dictatorial final call on what the main domain to use in any projects will be. The options are:
- harrisontemple.com
- theharrisontemple.com
- theharrisontemple.org
Im leaning towards the second option.
the second option as well. Just in case there's someone on this planet named Harrison Temple and wants a domain name 😄
Is the application done soon?
Sorry I was exhausted after a social event
There's like 6 hours of work left I recon
Doesn't sound that bad
Maybe I can do that all this evening after I get home
yea I'll need one more day to finish off navigation and backups
Ok, take all the time needed, I don't want a rushed application
Are you making progress on the application?
no. I will once I get back home
Ok then, I hope you get it done, but don't rush if you can't finish it
it's done ™️
(I will definitely want to make at least one more update)
but it's functional now
That's pretty good. Any updates on domain stuff?
And also when will we install it?
you can do that now
Is the process the same as the first time?
In what folder?
G:\hostStuff\neuro-rants
Is thiswhat it should look like?
Do I need to build the server?
Ok, what to do here
Alright
Seems like it runs now
Should I add the startup batch to the system's startup?
yea sure
It should now start with the system. What now? Is the domain still in-progress?
idk
Well, I haven't gotten updates and Rhae said they'd talk to you about something
^^
Well, none of those go anywhere at least
no I mean Rhae is looking for feedback on which one to use
Well, I'm fine with any of them
Or maybe one of the .com ones
Should I do anything else with the website itself?
you can check how things work
but no, nothing in terms of setup
we'll need to do something with certbot
you may want to look into that
What is that?
a software that we'll use to generate certificates
certificates are needed to encrypt the communication between server and client
that what the lock means
Oh ok, that seems important
I found some kind of instructions generator thing, what software is it running on?
The website
"Other"
So I just need to follow these instructions?
for now just the download and install stuff
the actual verification can be done only once the DNS will point at your IP
Ok I guess, I'll just download stuff then
It's installed now I guess
So on the website itself, what does the log in button do?
What is it logging in with?
discord
Oh, how convenient. Does it pull it from the app?
Does it pull the Discord login from the Discord application?
I don't know, I should probably test it
I guess it works
I'm gonna go to sleep for today, if anything happens, I'll check it tomorrow
ok, gn
Any updates on the domain stuff?
Any info on when we'll have the domain stuff functioning?
@sharp fiber ?
The server seems to have paged literally everything due to not being used
The server is now up and ready for testing: https://rants.theharrisontemple.com/ This thread will be used for technical updates, such as outages of the server. If you want to test it, feel free to do so, and please report any issues you find
Web site created using create-react-app
If you can't open it, the domain might not be on your nameserver yet, so try again later
We have data on the server now. Isn't that interesting?
I guess I'll have to add the other stuff sooner rather than later
other stuff being search, and moderation
We'll defenetaly need more tools for moderation. Apparently the date field is really hard to notice, so the data on there has an incorrect date value
Would be convenient to have a tool to be able to correct that
In case anyone noticed, there was an outage of unknown length due to Windows update automatically restarting and messing up some read permissions, while I was sleeping. This caused the server to go down, but it could not restart due to that read permissions error
It's back up now though
I was hoping to contribute to this a lot, but I am a bit curious about how this works
I did just try to publish one and it's not there quite yet (plus I missed the hiyori tag and would like to add that)
When you publish something, you will need to wait for moderators of the website (currently only me) to approve it
Moderator tools for editing website data are coming soon
Ah, I know one I should submit
Ah, thanks
That one was basically just throwing something in that hasn't been known about for a long time
But the other one submitted is certainly better well-known
I assume it's not supposed to look like this
yikes
It has a lot of tags that fit
There's probably a lot of bugs like this
That all looks correct beyond that bug though, right?
Yeah, it's just a formatting bug. Those will probably be fixed, since the current version is just a test version (all the data will carry over to the release)
yea thanks
Admittedly, I'm surprised a database of her rants hasn't come around before
But it's nice that we have something now
It's a pretty cool thing to be hosting too
People have made databases dedicated (or something similar) to specific types of Neuro quotes, but this website can basically fit all of them, since it has a tags system to seperate things like stories and poems
This one is also gonna be really easy to control, since it's hosted on hardware I have direct access to
True, true
And that's cool
It's also very convenient for me, seeing how much I go through clips to find stuff
I hope it's useful for everyone
Also, what are we counting as rants, in this case?
I'm sure it will
I say just collect interesting Neuro quotes and separate them with the appropriate tags
We already have a couple stories too
Ah, so it's not only for true rants specifically, but interesting quotes?
That's cool
I guess so, I'm not gonna run out of space any time soon
She does have some very funny answers as well. BTMC's meaning, for example
Should I add a TTS speaker to the website?
I'm not too sure honestly
It's not hard to add, I just need a color
Would say green because it's corpa, but Vedal is already green
twitch purple?
https://www.colorhexa.com/6441a5
I was thinking maybe
But my disdain of TTS was preventing me from suggesting purple, a colour I also associate with Neuro
I don't really know what color would be best, I can go with whatever
So what color should I use? I'll set up all the other data while I wait for a response
Well, I got all the other data for it in place now
Should I just use this for now, since it can be changhed later?
It is the colour of bits
Storing this stuff is really easy by the way, the current data is just under 5KB in size
a searchable data base would be amazing there are certain quotes i remember but I can't find them
A search feature is coming in the release updates
That is actually literally the original reason for the creation of this
Added "fact of the day" tag to the website
Sure, you can add whatever Neuro quotes you want
The data file is now a bit over 18KB. A pretty big increase
Huh, the double spaces disappeared from this one
Should be two spaces between "sauce" and "it's" and another two between "that" and "if"
I wonder if the website just can't display that for some reason, or if it strips that out of the data as "obvious typos" automatically
Hm, I'm not too sure
Well, I guess NotBot might know
We have a decent amount of data on the website already, which is pretty cool
And we haven't even released outside of very few places
Should I add a new "abber demon" speaker entry?
Added "chess" tag
I could probably use a different red of some kind for the abber demon speaker than the one Evil already uses, to make it easier to differentiate
if you haven't figured it out yet here is the answer
html basically ignores whitespace by default https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Whitespace
you want to add the white-space: pre or pre-wrap css styling depending on how you want it to look
https://developer.mozilla.org/en-US/docs/Web/CSS/white-space#pre
The presence of whitespace in the DOM can cause layout problems and make manipulation of the content tree difficult in unexpected ways, depending on where it is located. This article explores when difficulties can occur, and looks at what can be done to mitigate resulting problems.
I think this website is based on something called "Node.js" or something, no idea if the rendering is different there. I didn't make it, NotBot is the one responsibele for making the website and server, I just host it
node.js is just the server part of the website that sends the html. it is up to the browser to determine how to render the html.
I guess it's probably just ignoring the additional spaces then
yep that is how it is in browsers by default(first link)
We'll see if we do something about it
Note: Server outage scheduled after the stream today to install 64GB of RAM
Note: the server outage was moved to today, it is happening pretty soon
RAM upgraded, expect full server uptime for the near future
Ok, it's actually up now, I had some technical difficulties with my USB card, but that seems to be working now
im creating an OCR script for neuro's streams, sounds perfect to join this thread
sounds great
What does "OCR" mean?
optical character recognition
Oh, ok. Does that mean you could get text from the streams automatically? If so, that could be useful for getting data onto the website
Admittedly, I'm a bit hesitant to submit more before it's properly released
The data will stay the same even after the release. It would actually be beneficial to have more data before release
Also, for "the other day" category, do those words need to be said or can it be any story of something that happened to her in the past?
It's specifically for quotes beginning with "the other day", since those are quite common (there us one already)
New bug report:
The server is gonna go down for a bit, I'll reboot to fix Windows being absolutely broken and for setting fan curves
Server is back online now
Another bug found or something:
Semi related update: we have managed to feed Neuro’s rants in a LoRA! …now we need to convert it into a GGUF and we can have a “low budget neurosama” LLM
And it definitely only might just need a really high-end GPU to run it decently
This is how the Neuro Holy text is written
Neuro rants updated with tag filtering and moderation features
Personally I think they should also have numbers after each sentence to allow for better Neuro quotations.
You know how people quote the Bible? I thought it’d be funny if they labeled texts according to number after each sentence.
That sounds unnecessary
True
I don't think we want to unnecessarily add such unnecessary features, but add useful features instead, like a search bar (idea from someone in offline chat)
Was this moved to #1336777692070023271 ?
This is deprecated in favor of the quotes section of neurokaraoke.com