yeh
#databases
timber stratus
viral crag
It only does it once, and only after you push the task onto it
high gust
It's so much easier to explain asyncio with a visual reperentaion
timber stratus
screen share?
high gust
viral crag
Yeah, something like that
viral crag
Don't think too hard
Most people come into this stuff assuming it's difficult and they won't understand it
but really it's not that complicated
torn sphinx
what is wrong with user = sqlite3.connect('users.db') user.execute("UPDATE USER set subscribed = ? where ID = ?", (str(0), str(id))) user.commit()?
viral crag
Dunno, what's wrong with it?
timber stratus
yeah, what he said
elfin gulch
potent lintel
<@&267630620367257601> I need some help writing to specific parts of a notepad file
i have a user list in notepad
deft badge
And you want to edit one line?
potent lintel
each user separated by lines, and each piece of info (username, password, score) is separated by commas
no i want to edit one of the thnigs separated by the commas
high gust
Like a csv file?
deft badge
You'll have to split up the lines into an array and then each user into an array
potent lintel
yeah but i used txt
sacred heart
Isn't that a bit unsafe?
high gust
Why not use an acual database
deft badge
@sacred heart It doesn't really matter, that's like calling JSON unsafe
potent lintel
Reasons @high gust
torn sphinx
what if the username contains a comma
deft badge
@potent lintel can use whatever he wants, you guys are going to assist for his needs not tell him to use something else
potent lintel
i was initially going to use JSON but i can't
deft badge
@torn sphinx irrelevant
potent lintel
@torn sphinx i've validated that
deft badge
@potent lintel So what you'll need to do is get all the lines into an array
and then use line.split(",")
potent lintel
i've already done those two
deft badge
then you'll have an array of the values of one line
deft badge
Yeah edit whatever you want and then use file.writelines(array)
potent lintel
how do i specify which line
potent lintel
oh
deft badge
Do you mean which line to edit?
potent lintel
No i can already get that
deft badge
Ahh okay
well yeah just split into an array, edit a value and then put back into a larger array of lines
then file.writelines(array)
https://stackoverflow.com/a/4719562 like this a little bit
deft badge
No problem
blissful tapir
@deft badge I swear f.writelines doesn't actually exist
torn sphinx
deft badge
t = open("f.txt")
torn sphinx
deft badge
oh
blissful tapir
gg.
torn sphinx
deft badge
t.close()
deft badge
lmao
blissful tapir
e.e
near cradle
:D
dusty wolf
Does anyone know of an example of Python, MongoDb and Cisco switches?
tranquil delta
@dusty wolf never used a Cisco Switch, but Python and Mongo is simple enough. Though, know your needs because Mongo isn’t and never will be ACID-compliant
crystal lion
oze - Today at 11:39 AM
oioi
I have a rather general question:
I have existing data and want to import it into a fresh SQL-Alchemy/Flask database
those data-sets already have a unique ID, i.e. primary-key
how do I ensure that the existing ids/primarykeys are used in my table as actualy primary-keys for my db
is setting auto-increment to "off"
for my fresh primary-key/integer column, i.e. table in my SQL db-model...
enough?
*actual
oh yeah: possibly relevant .. the existing ids/primarykeys do not start at 1
onyx gate
I cannot import PyMongo because it couldn't be possible to find flask.ext.pymongo first, can anyone tell me why?
viral crag
Proably because you don't have flask.ext.pymongo
onyx gate
Isn't downloading the package flask enough?
bright pelican
👍
onyx gate
And how can I let the system read a single line of a database document?
Like:
[1] a
[2] b
[3] c
and I want the system to read only the line 2
ornate abyss
@onyx gate This issue is very old, but did you ever fix it?
onyx gate
Hey! Yes, I fixed this but there are some things that I would like to get help with
@ornate abyss
torn sphinx
Hi everyone! Before I start reinventing the wheel, I want to check if there's an existing Python library to do XMLSQL web queries. I Googled and it doesn't seem to be a standard? Basically, sending SQL queries over XML via forms.
So a sample query would be like this:
<QUERY metadata='true'>
<SQL>SELECT personid, name, lasttransdate, binaryobjectid, payfeeatpof from person
where name='Smith'</SQL>
</QUERY>
torn sphinx
a client request, they have this rather odd server, that i'm wondering if they wrote from scratch, or they based off some library they found in Java or something
viral crag
It's not something I've ever heard of, nor is it something I can find any mentions of
torn sphinx
if I can't find a library, i'll just have to roll out my own Python implementation to send their XMLSQL requests
viral crag
I'd say you might be on your own with that one
Is that actually sending SQL directly to the database?
Because you really don't want to be running SQL from a user..
torn sphinx
oh this is an internal facing app
torn sphinx
the computers might even just be connected to each other directly via ethernet, heh. yeah relatively ancient API, the docs say it was invented 6 years ago
viral crag
Hm, json was still popular back then
Anyway yeah, good luck, sounds like you're gonna need it
torn sphinx
man, I wish it was JSON
viral crag
When u worked for fexco, they actually had me translate a Western Union backend app into node as a proof of concept
I had to write this thing to translate their XML into json
The whole office uses json now lol
torn sphinx
heh, the wild wild west of ancient companies in ancient industries
viral crag
Indeed
obtuse lance
hola. I'm trying to figure out what kind of system I'm trying to build, or if there's a name for it; or whether I'm doing a bad thing.
I have a bunch of nodes which each produce records of contacts (myname, theirname, time). These records all exist in the same universe/scope/table.
... (see lower message)
wispy fable
@obtuse lance You only need to store a table with those values?
obtuse lance
Yes, I only need to store those values; doesn't even need to be a table, and the time isn't reliable, so storing it is optional.
(I related it to tables/SQL because that's the vocabulary I have on hand.)
The DAG thing I imagined was a directory+file structure, files named $theirname with contents $myname $time.
wispy fable
So you're mostly concerned about syncing?
wispy fable
Have you considered writing your own program for this?
obtuse lance
Yes, which is why I'm happy to accept relevant reading material :p
I dug into doing it something similar at one point and got way over my head very quickly
wispy fable
Well, I don't know much about databases but I do have suggestions for keeping multiple machines in sync
obtuse lance
(inotify+cron)+rsync? ;p
wispy fable
I was going to suggest writing your own syncing code, but rsync would be better for the actual transfer of data
obtuse lance
correction: probably will need to write my own implementation because the target audience.
so yeah, I go directly back to DAGs and stuff
rsync seems like a run-away idea
wispy fable
your own implementation should work fine as long as they aren't changing the clocks on their machines constantly
I was going to recommend that a machine checks for other machines on the network, then checks offset between the network clock and their own clock and fixes that, then uses the adjusted time for checking sync
obtuse lance
right, so, you're losing me.
I'm not any closer to understanding how to do that
also, no. Clocks are like timezones, I'm not fucking touching them with Dundee's knife if I don't have to.
wispy fable
Will entries ever be removed from the list?
obtuse lance
yes (but in a DAG it will be another node appended to the tree so it won't matter?)
wispy fable
I can sorta do databases, but I have zero experience in networking, so I doubt my ability to help you
obtuse lance
oh well, thanks for the time ;p
wispy fable
obtuse lance
repeats down here
hola. I'm trying to figure out what kind of system I'm trying to build, or if there's a name for it; or whether I'm doing a bad thing.
I have a bunch of nodes which each produce records of contacts (myname, theirname, time). These records all exist in the same universe/scope/table.
I very very much want to avoid any centralized database or control because the target audience doesn't know how to keep track of the central machine. I'm uninterested in "security" because of the audience.
I can't rely on machines having reliable clocks, at all. I have also found that I can't use leaderless consensus because nodes will appear and disappear, and I cannot guarantee that any number of nodes will be online or that they won't otherwise meet in other network partitions.
I have thought about using git/DAG model: Each node is the owner of a DAG. Any node can provide updates for any other node. It avoids needing clocks to order events.
Using different DAGs means I would have to "coalesce" all trees to create the "universe/table," but I believe that is a very lightweight operation. Conflicts aren't a problem for me in this case.
I'm not sure what this is called, or whether there are downsides to this option, or whether it's realistic.
I would appreciate advice, or a general idea of what relevant stuff I should read up on :D
viral crag
Perhaps take a look at the "spanning tree" model
Another option would be to look at how MongoDB does replica sets
obtuse lance
Spanning tree model for syncing data? I don't need to ensure efficient sync, so that seems like run-away, orthogonal, unnecessary complexity.
viral crag
You lost the opportunity to avoid complexity by having requirements that prevent you from having a central node
obtuse lance
Certainly, however the client can use multicast without caring about STP. Can you help me understand what about this complexity makes it necessary in this situation?
viral crag
You don't have a reliable clock
If you had a central node, that wouldn't matter at all
You can just get the time from it
I'm not really sure what audience you're working with either
Not caring about security suggests that they have the expertise to set up a closed intranet for their nodes
But you suggest that that isn't the case by saying they can't take care of a central node
obtuse lance
Their needs are simple and security is simply not one of them. They are not experts of any kind :p
I'm not with you on why not having a clock leads to STP. Can you take me there?
viral crag
Well, I didn't actually tell you to use STP, I suggested some research possibilities :P
but not having reliable time is very.. 90s
any kind of crypto is going to need that, as well
obtuse lance
I'm so confused by what you're thinking of.
There's no crypto involved whatsoever
DAGs (if you're talking about that) generate hashes, which don't rely on clocks, much less sync'd clocks.
And I still don't understand what aspect of STP you're suggesting is relevant to this situation? :/
viral crag
The problem is that I don't really understand your requirements, and you may be expecting a full-on solution as a suggestion here
the reality though is that for something like this, the most you're gonna get from most developers are semi-relevant topics to do research on
because the situation you're describing appears to be incredibly niche
You also seem to already have some kind of implementation detail in mind
obtuse lance
Right, I do have a possible implementation in mind, because an implementation is my goal; given the restrictions I'm aware of :p
I am trying to work backwards and forwards because I feel that I'm missing some fundamentals to know what it is that I'm building.
I don't understand how you connect STP and this situation. In what way is STP semi-relevant to this situation or potential implementation?
I can learn from that answer. I can't really learn anything useful from reading the STP RFC with no goal.
viral crag
The point is that you need some way to get every node connected together
In most situations, having every node update every other node is very inefficient and could slow the application down
having some kind of hierarchy is important in that case
obtuse lance
premature optimization.
viral crag
You're telling me you'd be happy to rewrite the entirety of your networking code later? :P
viral crag
Are you stuck on python 2?
obtuse lance
Like I said, I'm not sure how, or if, STP helps me understand a fundamental design idea which I could use for syncing a bunch of records produced by byzantinely available, independent nodes.
viral crag
Then don't use it, I guess
obtuse lance
I'm asking what about it you connected to this situation
It's a confusing suggestion to me
viral crag
I mean, I explained as best I could
obtuse lance
At this point I'm really curious about the reasons you went to STP?
viral crag
I don't really think I can explain it any better than I have, sorry
obtuse lance
okay :p
wispy fable
@obtuse lance Will the number of machines in use change?
obtuse lance
yes
(so precluding raft/paxos ;p)
I might be super dumb. If each node is the author of its own records, and I'm just coalescing the records from each node to create the "universe," I don't need trees or dags to keep history. I can use a per-node version number and always prefer the higher one when syncing.
Sweet, this got easier for me
wispy fable
here's some pseudocode I made
#retrieves locally stored ledger of machines to connect to as well as a locally stored version of the info to be shared
ML = GetLocalMachineList()
L = GetLocalInfo()
#Attempt connecting to each machine then retrieve a global list of connected machines when a successful connection occurs and update it with itself.
#Also gets a current version of the list
for machine in ML:
if AttemptConnection(machine):
ML = GetMachineList(machine) + GetSelf()
GetInfo(L)
Pushupdate(ML)
break
I appreciate it :D
wispy fable
Again, I have zero networking experience
There would also be removal of machines who disconnect for too long, and a backlog of all connected machines.
I forgot to do a clock thing
I did have something on mind
also keep in mind that ML is sorted from oldest to newest in terms of connection time.
#requests the global time from the oldest machine, then generates an offset amount. RequestTime is ping adjusted as well.
GL = time.time() - RequestTime(ML[0])
def Time():
time.time - GL
catalogue = { 'node_id1': Logs(version=10, data=...) , 'node_id2': Logs(version=187, data=...), ... }
sock = socket.open($multicast)
while 1 {
recv = sock.recv()
if recv:
recv = process_magically(recv)
if recv.type is ADVERT and catalogue[recv.node.version] < recv.version:
sock.send(build_request(recv.node))
else if recv.type is REQUEST and catalogue[recv.node.version] > recv.version:
sock.send(send_log(recv.node))
else if recv.type is LOG and catalogue[recv.node.version] < recv.version:
catalogue[recv.node] = Logs(version=recv.version, data=recv.data)
sock.send(advertise(catalogue))
time.sleep(1)
}
an advertisement will cause a flood of requests and subsequent flood of logs, but this is the basic idea I have now
thoughts?
Feels like I might be reinventing a wheel
wispy fable
instead of sharing the information directly, why not share a list of changes. That way, removal of entries and simply being out of date would be separated
You most likely are.
obtuse lance
Because diffs are hard and add complication :p
I could do it if trees, but I might as well just cache the last 10% of versions to do diffs against.
LOG_DIFF packets would be easy to add
Yeah, so I'm curious if there is a name for what I'm trying to do :p
wispy fable
blockchain
wispy fable
it's a decentralized ledger
obtuse lance
Requires a dedicated number of nodes and is a mitigation against lack of trust
I can't guarantee enough online nodes to prevent a split, and I have trust
wispy fable
you said you didn't care about security
There might be a way to adjust behaviour to prevent splits
obtuse lance
Right, I can trust any potential node, so I don't care about security
Modifying blockchain implementations to avoid forks seems super scary, especially since my understanding of how it deals with forks is to discard all but one by design.
wispy fable
it was partially a joke
but you should try searching with the keyword "decentralized"
wispy fable
Cool!
warm hare
MongoDB or PostgreSQL?
What would ya guys recommend
Its about an application that's currently relatively small
but we would prefer to not ever have to move to a different DB
isn't mongoDB sharding hellish or smth?
deft badge
Not needed in most scenarios
warm hare
hm so what would you recommend?
deft badge
Out of the two, I like Mongo, but that's personal preference
I use RethinkDB primarily and Mongo is closest to that
shut crater
Anyone know a good step-by-step or tutorial for MySQL and Flask? I'm comfortable with python and have a fair grasp on flask now. I know SQL but only have used MS access.
broken linden
you could go for https://www.sqlalchemy.org/ or https://github.com/PyMySQL/PyMySQL they are the two most poular for that i guess, although i would recommend to at least go for postgresql and https://pypi.python.org/pypi/psycopg2/2.0.4
shut crater
Thanks, I'll take a look
obtuse lance
@warm hare use an ORM so you don't have to worry about switching databases.
SQLAlchemy for Python, PHP has one built in, Diesel for Rust, ActiveRecord for Ruby
warm hare
there's an orm for like every DB nowadays
obtuse lance
Pick one. It's not like your choice of database is important to you since you just asked whether you should use <Document> vs <RDBMS> without outlining what your data looks like, so maybe you shouldn't use a database at all.
At the very least, an ORM leaves you in a position to make a less invasive change if you find out the approach to databases you took was the wrong one.
ionic pecan
is it possible to generate .sql scripts from sqlalchemy somehow?
ionic pecan
sorry, I should've been more specific, I meant doing this but for creating tables depending on dialect
because you create tables through Base.metadata.create_all(engine)
obtuse lance
Does the last section not meet those needs?
"If you want to take a peek at what is going to be sent to the DB, you need to tell the SQLAlchemy compiler what dialect it should be using. Here’s a way to print exactly how it’s going to be compiled to:"
I'm not familiar enough with it to know, but I imagine the resulting CREATE can be inspected as well.
ionic pecan
Hmmm, I‘ll see what I can do
obtuse lance
@ionic pecan
Not sure how helpful that link is, though
It's horribly formatted on mobile
ionic pecan
Thanks, thats just what I needed!
obtuse lance
neat!
tropic zealot
Are there any major differences between SQL Alchemy and SQLite3?
I wanna start getting into usage of SQL, but don't really know where to start.
There are also no pinned messages, so I can't check those for information
Alright, after literally 3 minutes of searching, I've found out that they're almost 2 different things.
A better question now would be to ask if I should just learn how to use SQLite3, or if I should incorporate SQL Alchemy into my learning of SQL.
broken linden
first learn the basics of the language then think about how you will use it
but for the learning process itself sqlite would prbly be easier to use
tropic zealot
Thank you very much.
viral crag
Sqlalchemy is an ORM
You write code using objects and query those objects, and Sqlalchemy takes care of the SQL for you
Which means that you can swap out the database type at any point
trail imp
Anyone got a good resource recommendations for getting started working the python and mysql?
broken linden
Uuh .. Dont work with mysql use postgres and for that u could take a look at psycopg2 or sqlalchemy. if u still want to start with mysql use pymysql or sqlalchemy
@trail imp
trail imp
Alright, ty. Why are you saying not to work with mysql?
obtuse lance
Oh, dude, so many reasons. Mostly silent data loss. Nonstandard behavior is a legacy of mysql.
broken linden
^^^^
obtuse lance
Change your schema? We'll fill in your new NOT NULL column with system default data for you! We're so helpful for filling out your age field for you.
Did something go wrong in this transaction involving your schema change? We'll just silently commit it.
It's fucked. It's so fucked.
They've done a lot of work to improve the situation. I don't think they just silently truncate data any more if you insert longer decimals than the schema says it stores, but there's plenty of reason to prefer postres or maria.
ornate abyss
@tropic zealot If you want to learn SQL itself dont use SQLAlchemy, the point of the ORM is to handle things through objects and methods and not actually write SQL itself
tropic zealot
...that was from about a week ago, and I already got what I needed explained to me...
spark matrix
lmao
ornate abyss
Oh shit true, it wasnt scrolled down so i didnt notice
You could have said it in a nicer way however, i was only trying to help
obtuse lance
^
wispy fable
still useful to have a tl;dr
torn sphinx
I have a function in postgres that has return type table. This table returns rows of username, firstname, lastname, sessions etc (username is in a different table). I need the column names and data type of the return type table. So that I can use those column name and datatype inside django rest framework. Any idea how can I get the column name and type of a function that returns a table?
hexed mango
anyone here work with Mongo need some help
hexed mango
I am running into an issue where If I want to insert 100 documents into 100 different respective collections, so one doc per collection. The doc size is not huge either about 10 fields and none of them are huge largest value is the unix timestamp. the run time to do the 100 inserts is taking about 30-40 seconds it seems to be way too long. I am curious if this is even the correct way to be using mongo, or is there a way to like commit the cursor like in sqlite to eleviate the call stack.
I am not doing anything that hasn't been done before, it just seems like the driver is struggling
ionic pecan
@ornate abyss knows mongo
hexed mango
ty @ionic pecan
from pymongo import MongoClient
data = urlrequest // 100 item json
client = MongoClient()
db = client['test-database']
for i in xrange(len(data)):
db[data[i]['name']].insert_one(data[i])
thats basically it
hexed mango
Problem is solved, it was my inexperience with mongo turns out initializing the collections for the first time takes some extra time I wasn't expecting.
tropic zealot
I have some data that I'm gonna be updating and using in a Discord bot that I'm writing for my friends and I. This data is going to be containing things like D&D character sheets. I have a format in JSON already created, but I was just wondering just to wonder:
Would it be better for me to move to SQLite3 for something like this or to stick to JSON? IIRC, we're gonna have about 10-15 people playing max.
Just for reference, here's what I have in JSON:
{
"name": "null",
"age": 0,
"race": "null",
"class": "null",
"religion": "null",
"stats": [
{
"strength": 0,
"dexterity": 0,
"constitution": 0,
"intelligence": 0,
"wisdom": 0,
"charisma": 0,
"luck": 0,
"speed": 0
}
],
"experience": 0,
"level": 0,
"background": "null",
"alignment": "null",
"languages": [],
"hit_dice": "null",
"hit_points": 0,
"proficiencies": [],
"saving_throws": [],
"abilities": [],
"skills": [],
"gold": 0,
"initiative": 0,
"perception": 0
}
using flat-files to do data storage, manipulation, and recall is reinventing the wheel.
I doubt you're going to suffer for having the extra weight of an sqlite import, and it's going to make far better choices than you would about the data
It isn't necessary. You can run facebook on flat files if you're really really determined; but it's not like you gain anything from avoiding the extra bit of work to interact with sqlite, which means you now get a decent effort at protecting your database from application crashes for free.
viral crag
to be clear
sqlite is still flatfiles
unless you're using an in-memory database for whatever reason
but yes, it's certainly a better option than rolling your own
obtuse lance
sqlite store is a filesystem.
You can say your rootfs is also a flatfile since you can just copy it into a file or so, but just because it's in a file there doesn't mean it's not still a filesystem :/
viral crag
the difference between that and rootfs is the kernel has support for rootfs
obtuse lance
wat
viral crag
well, unless you're on windows
obtuse lance
As if FUSE fs's userspace filesystems aren't "real" filesystems, bro?
viral crag
I mean, they're not, technically, but they're as good as
(EDIT: Message was edited after I replied to it)
obtuse lance
No, dude, they are.
viral crag
sqlite is not implemented that way either way, if memory serves
obtuse lance
Filesystem is a specific structure, it's not decided by whether the kernel supports it.
viral crag
I don't think sqlite is a filesystem whatever way you spin it, honestly
viral crag
I'm pretty sure it's just a paged/segmented file, no?
that's what the documentation says, assuming sqlite.org is the site to find that
...
The main database file consists of one or more pages. The size of a page is a power of two between 512 and 65536 inclusive. All pages within the same database are the same size. The page size for a database file is determined by the 2-byte integer located at an offset of 16 bytes from the beginning of the database file.
filesystems by definition are systems of files
pages are not files
by that logic, any segmented file is a filesystem
my GIF avatar is a filesystem
yeah, not once does the sqlite documentation call the format a "filesystem" either
viral crag
If you're not willing to explain yourself, don't make the assertion lol
I looked up the documentation, that's what it says
Why do you feel that sqlite is a filesystem?
tropic zealot
...so I'm guessing I should attempt to migrate over to SQLite?
obtuse lance
You're not willing to explain your ridiculous idea that what qualifies something as a filesystem is whether it's supported by the kernel. You have no room to talk.
tropic zealot
This entire debacle sparked off of me asking a question.
viral crag
I later corrected myself by giving the definition of a filesystem
obtuse lance
If that's how you think conversations work 🤷
viral crag
You're the only one being confrontational here.
obtuse lance
You're the only one being frustrating here
viral crag
You need to take a step back if you can't control your own mood over something as trivial as this.
obtuse lance
(I even already left this alone. What are you on about anyway.)
viral crag
It's a simple question
obtuse lance
No
viral crag
You seen convinced that sqlite is a filesystem - why?
obtuse lance
This is silly.
Hmmm?
Wait
Are you interpreting that in the least charitable, most literal sense? Is your entire "just a question" based on a forced misunderstand? From the guy who said that SQLite is flat files comes the interpretation "databases are filesystems"?
viral crag
You made that assertion, but from what I can find, a filesystem is simply an abstraction of data using "files", and the sqlite file format documentation doesn't mention it
So I'm curious, it feels like I'm missing something
viral crag
Programmers argue about semantics all the time :P
tropic zealot
It's barely even related to what I asked 😛
viral crag
Well, the answer is yes, you should move from what you have to sqlite
Our discussion is separate mostly :P
obtuse lance
Our discussion argument is entirely irrelevant to your concern.
You should use sqlite for the reasons outlined 2 buffers ago
tulip hare
upps wrong channel
viral crag
No, that is the opposite
tulip hare
but the part about getting django models to be used in scrapy
import os
import sys
# DJANGO INTEGRATION
sys.path.append(os.path.dirname(os.path.abspath('.')))
# Do not forget the change iCrawler part based on your project name
os.environ['DJANGO_SETTINGS_MODULE'] = 'iCrawler.settings'
# This is required only if Django Version > 1.8
import django
django.setup()
# DJANGO INTEGRATION
## Rest of settings are below ...
wouldnt be a good solution at the point where i setup the website on a server?
it appears to me that this solution is way easier so i could handle all my Database communication inside each Spider
but i might be missing a crucial part of possible problems later on
@viral crag
is it cuz i should not interfere with the django.setup() while its running
or that i should always avoid doing this to django?
torn sphinx
I'm trying to complete sqlites tutorial didn't understand this part https://vgy.me/ZwOOmj.png
How do i actually connect to the database?
Do i have to add sqlite3 to the PATH?
ionic pecan
don't think sqlite has their own GUI tool
that was linked in the manpage, I suppose that's supplied when you have sqlite installed
torn sphinx
db browser for sqlite looks simpler ill go with it
torn sphinx
Cardium requires a gui. For now
ornate abyss
Did you just talk in third person
tropic zealot
On a scale of slow to fast, what is the speed in which Python reads through .db files with SQLite3?
near cradle
that's shitty scale. why don't you time it?
tropic zealot
🤔
I don't think I follow.
I also don't think I've started with my database, yet, which is why I'm asking about speed now.
viral crag
to give you an answer that makes the same amount of sense as your question:
"Eeeeeeeh."
tropic zealot
Okay...
I'm also gonna guess that lemon was referring to this https://docs.python.org/3/library/timeit.html
when he referred to "timing it"
vestal apex
it's a popular module
dense shadow
what are the disadvantages of pickling a dictionary for data storage over traditional databases such as SQL, aside from compatibility?
viral crag
Well dicts aren't implicitly persisted
the comparison doesn't really make sense on its own
dense shadow
implicitly persisted?
viral crag
I mean, the whole point of a database, surely, is that it's saved
it's not just stored in memory
dense shadow
I ask this question because I witnessed an argument over what someone should be using for their large-scale web application in python
dense shadow
ah, so instead of loading the whole thing from disk it's just fetching sections in sql?
I'm not too familiar with the methods that sql uses myself
they'd mentioned having the option of pickling a plain dict, using CSV, or using SQL
viral crag
I mean, large-scale pretty much requires a database for most tasks
You can't just have a single process with a dict in it
vestal apex
pickling and CSV creation are just means of dumping a Python object into a file
because those objects can get large, and to avoid the issue of needing to manually dump periodically to keep your data safe, you use database software to curate efficient access to it
SQL stands for Structured Query Language
it's a language commonly used to access relational databases
not any particular piece of software
so like G said, databases are kinda required for large scale applications with lotsa data
file dumping gets problematic quickly
raw zenith
What's the best way to schedule an event for a potgres db? Basically I need to trigger a function of some sort after an INSERT into a certain table that will send a text (Twillio) two hours after the INSERT took place. (I'm using psycopg2)
tulip parcel
@raw zenith
Honestly, the easiest and fast way is a cron job that runs every 5 minutes. Make sure your table has a column "notify" and "date created" so that the script knows when to send a notification.
When your cron job runs it selects all rows notify=0 and date_created more than 2 hours ago. after it sends the notification it sets notify to 1 (or true)
Heavy handed way is to use celery or python-rq
sterile thunder
im trying to make a monitor and its done but im stuck on the final thing creating and maintaining a databse, so i made one in json and made it so that it checks if its new data or not, now how do i append the database file
tidal leaf
hello
sterile thunder
nvm
bright pelican
b
torn sphinx
c.execute("INSERT INTO mesajlar VALUES(message.content, message.author, message.timestamp, message.author, message.author.top_role.name, message.author.voice_channel)")
sqlite3.OperationalError: near ".": syntax error```
Can you guys see anything out of order?yup
you need to pass the message attributes separately
sqlite doesn't know what message is
c.execute('INSERT INTO table VALUES(?, ?)', ('dank', 'memes'))
so i write message and stuff on right?
torn sphinx
thanks that did it
torn sphinx
btw
do a search on that page for 'parameter substitution'
that should be standard for any sqlite insert that uses any user submitted content
the questions marks aren't just placeholders for variables
oh so now im secure
wow and im logging messages directly someone could mess up easily
i wouldn't go that far, but now you're more secure than just passing user content
right
ala "sql injection"
bright pelican
oh god
you're logging user message content too
good thing parameter substitution was suggested
torn sphinx
yeah im not planning to use it for long but i could use something in the future and fuck up big time
should i assign them to a variable first as its suggested or is it the same thing?
bright pelican
Using ? parameter substitution is fine, yeah
torn sphinx
❤ thank you
mossy light
anyone ever have issues with aiomysql and queries wrongly returning no results like half of the time?
ive checked my db through mysql workbench and used the same sql queries and got the correct results 100% of the time
im using a connection pool with autocommit set to false
setting autocommit to true seems like it might be fixing the issue but it also slows down performance a ton
ornate abyss
What do you mean it's wrong returning no results
Do you mean it's not returning data you recently inserted?
mossy light
It will return no tuples despite there being a matching record in the database. If I copy the query in the code and run it in MySQL work bench, I get the right result
tropic zealot
I've been looking back into databases again (seeing as how I'm going to be using one for my Discord bot), and have come to the conclusion that I'm going to have to use eithr MySQL or PostgreSQL in order to perform my database queries. I, however, don't know which to go to. Do any of you have suggestions that would lead me to one or the other, based on the databases themselves, or the async modules that they run off of in Python?
gusty spindle
i've heard that you should use sqlalchemy-aio with discord bots
tropic zealot
I've heard around the grape vine that people shouldn't really learn SQLAlchemy if they don't know SQL, since it deprives them of other useful knowledge. I'll take it into consideration and look into it, but I'm just putting that out there.
gusty spindle
alrighty
viral crag
SQLAlchemy is not yet suitable for asyncio
I would suggest using a database that has a driver that is, such as rethinkdb
viral crag
Not much but yeah
ionic pecan
a b s t r a c t i o n s
👍
ornate abyss
@tropic zealot a choice between MySQL and postgres go with postgres
ornate abyss
Why do you need to use NoSQL?
tropic zealot
NoSQL?
I don't know what that is.
I'm looking for a database to store user information that I'm trying to collect. Don't want multiple JSON files in a storage bin, so I figured a database would be nice. Need something Async, which means only certain kinds of databases, along with their async libraries, will workfor the project.
I might just try rethink and move onto something else if I don't like it.
ornate abyss
NoSQL is a term used for when the database doesn't use SQL and is non-relational
shy magnet
Does anyone know if its possible to INSERT and SELECT in the same query with pymysql? I can execute a query like
SELECT LAST_INSERT_ID()```
in mysql workbench, and I get the return value I expect. If I execute the same query in pymysql through the cursor, nothing seems to be returnedI was also considering switching my program to use postgres. Decided not to since it seems a like a bit more work, and I can accomplish what I need with mysql. If this is a solution that can be more easily solved with postgres, I'll probably switch since I will need to tackle this on multiple occasions
near cradle
you do terminate the second line with a semicolon too, right? SQL likes semicolons.
shy magnet
added the extra semi, still no luck
near cradle
also this isn't really doing insert and select in the same query, this is doing them in two separate queries. any database should be able to do that, but I have never actually used pymysql so I don't know how particular it is about that sort of thing. it strikes me as odd that it wouldn't work.
have you gotten any query to work so far using pymysql?
shy magnet
Yeah. if I execute that query, the insert is successful if I check the table, just the return doesn't work
near cradle
can you return without the insert?
perhaps you're just not reading the return data properly?
shy magnet
Im just printing the raw cursor.fetchall()
seems to give me the right data on other queries
near cradle
right, so it's just these multiqueries that are turning up blank?
hmmmm.
@shy magnet apparently there's a executemany method
try that.
shy magnet
Ill check it out. thanks lemon
near cradle
no problem, guy
torn sphinx
I'm using js not python, but with sql-ing, what's wrong with "UPDATE USERS SET Name='" + String(newname) + "' WHERE UserID='" + String(userId) + "'"
ionic pecan
not using parameter substitution
unless String performs some cleanup
maybe missing a semicolon?
viral crag
@torn sphinx Don't just insert strings like that, you're leaving yourself open to SQL injection exploits
torn sphinx
in js sql injectsion exploits don't happen from inserting strings like that
not through mysql
torn sphinx
equivalent of str()
ionic pecan
yeah
torn sphinx
thanks!
ionic pecan
i don't know which sql library you're using but you should be using parameter substitution for this
torn sphinx
ok...
im using mysql.. but i'll find a colution
*solution
now no exploits: newname.replace("'", "").replace('"', "")
well u know
newname.replace("'", "").replace('"', "")```@torn sphinx Don't be that guy
Do it properly using the parameter substitution of the library
Or don't do it at all
viral crag
No, it doesn't.
torn sphinx
why not?
viral crag
Your attitude is appalling mate, there's no reason for you to think you know better than the library devs when it comes to security
torn sphinx
no i know im not better
viral crag
This is a completely solved problem
And people doing it themselves is exactly why we have problems
viral crag
The library will solve this problem for you.
torn sphinx
no i know you can do it with the library
viral crag
And it's far simpler than what you're doing, most likely
torn sphinx
but it's just more effort and complex
viral crag
If spending 10 minutes reading the docs is too much work, then you're in the wrong field, mate
torn sphinx
look i think im fine, if someone really wants to exploit me they can
i really am not that uptight about security
thats all
viral crag
You should be.
torn sphinx
but it's not that important
viral crag
There is a current epidemic of software that people are using with these kinds of exploits
They're rookie mistakes
But they are making the overall software scene very tricky to deal with
viral crag
People don't know who to trust, and developers are finding it harder to get their software out there as a result
Yes, many projects are written by hobbyist developers
torn sphinx
look, it won't make any difference to the world of devs
viral crag
Probably a majority tbh
torn sphinx
i just do this for fun, spare time, barely do anything with it
viral crag
And if/when you get used to it, make your own public project or get a job in it
The fact that you have no idea how to do it correctly will come back to bite us all
torn sphinx
it is public but it's just a discord bot
viral crag
I'm not just talking out of my ass by the way, this is a thing that happens
If you're gonna learn to do something, do it right the first time
That's all
torn sphinx
i mean if someone want's to change their name just to exploit my bot idc...
i understand
i am doing it wrong...
but it's fine at this level
for example my other project, which is bigger and in python, does do it the "right" method
i just don't want to be doing that with every single thing
viral crag
Sorry, but I would not use your bot in this case
viral crag
When someone could change their name and potentilally delete or leak my data
I'm sure many people would feel the same
torn sphinx
but they can't really..
ionic pecan
what am I reading
torn sphinx
they can't "leak" your data
viral crag
Text on discord
ionic pecan
in the time you've spent discussing this you could've already changed your code to just work the right way multiple times
torn sphinx
i am using some basic safety: .replace which is fine by me
ionic pecan
that's not "basic safety"
torn sphinx
no i can't it takes ages to restart
ionic pecan
what I gave was just a dumb example
just find (whatever mysql library you're using)'s docs and find parameter substitution
ornate abyss
Use pymysql
viral crag
Friends don't let friends write insecure code
ornate abyss
ive used it before its nice
torn sphinx
im doing it in js, but thanks
ornate abyss
what the fuck
viral crag
@ornate abyss knock it off
ornate abyss
knock what off
torn sphinx
lol, calm down
ornate abyss
i aint angry dw
torn sphinx
ok u know, what ill use parametrised thing
but ill only restart like tomorrow
happy then @viral crag @ionic pecan ?
ornate abyss
im just confused as to if you're using js why did this conversation even start
ionic pecan
it's not my app that's vulnerable, so I don't really care. i'd be happy if you would see why it's useful instead of doing it because we've been telling you to
torn sphinx
well, i understand it's security reasons
but i don't see what more they'll do than replace the ' and ", but i guess im not a big dev
im using that
ornate abyss
Using a prepared query means only one keyword will be executed, if you do your method, thats simply a work-around, meaning that it can still be exploited via other means
torn sphinx
let q = "UPDATE USERS SET Name= ? WHERE UserID= ? "
var query = connection.query(q, [newname, userId], function (err, results, fields) {
if (err) throw err;
});``` there donethat didnt take long
torn sphinx
also no passwords are stored
also it takes about 15 mins where my bot is offline now
trying to turn on
ornate abyss
it takes 15 minutes to turn on?
torn sphinx
yea the host restarts
finds a server and stuff
sql is just so bad in general
it should be like changed completely
the concept
although, i don't offer any btter solution
bcz im me and im bad
i hv exams in a week, and here i am editing my bpt
*bot
ornate abyss
SQL is very good tho
viral crag
Both GDPR and the discord terms of service smile upon you @torn sphinx
torn sphinx
lol 😄
near cradle
oh dear.
viral crag
Haha, hi lemon
near cradle
hi 
4 hour discussion on whether or not to use parameter substitution in SQL.
this is why we can't have nice things.
ornate abyss
:^)
unreal tartan
"whether or not to use parameter substitution in SQL"
What's the alternative?
dusty scroll
This is parameter substitution right?
INSERT INTO <table name> VALUES (%s, %s)
then cursor.execute(sql, (value one, value two))
Just wanna make sure
unreal tartan
I'd say so, yes.
dusty scroll
Ok thanks
rain karma
"Linear search and binary search are the basic algorithms for implementing select operation. If the tuples of a relation are stored together in one file, these algorithms can be used to implement the select operation. However, if indexes such as primary, clustering, or secondary indexes are available on the attributes involved in the selection condition, the index can be directly used to retrieve the desired record(s)."
this is from my database book
what does it mean when it says "indexes such as primary, clustering, or secondary indexes are available on the attributes involved in the selection condition,"
ornate abyss
Im assuming this is answered since you asked the same question in the other discord
spice drift
anyone got any idea of how to return the position of a specific entry? pretty new to python and sql
but say i have 500 entries with unix timestamps and i have an identifier for each
i want to find the "rank" of that unix timestamp by using the identifier and descending/ascending order
how would i do it?
spice drift
right so i'm setting up a discord bot linked to a sqlite3 database
viral crag
if it's an SQL-based database, you should have access to ORDER BY
so eg ORDER BY column ASCENDING (or DESCENDING)
that's as part of a larger SELECT query
spice drift
gotcha
it's just whenever I use select I only know how to set it up so it returns a single value
viral crag
so eg, something like SELECT id FROM tablename ORDER BY column ASCENDING
you should end up with a cursor object you can iterate over with a for loop, or you can exhaust the iterator into a list by list(cursor)ing it
spice drift
i dig the idea of the for loop
viral crag
so that'd be something like for row in cursor:
spice drift
take all entries, for each one add 1 to a value, and compare it to the requested ID
then i have the position when the for loop actually finds the ID
thanks man, appreciate the help 😄
viral crag
No worries
gilded jungle
Hey Pythonistas, I'm new here so Hi everyone !
I'm really needing help on an sqlalchemy-graphene problem, is there someone here that could help ?
Here's my database automapping :
engine = create_engine(" blablabla", convert_unicode=True)
db_session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
Base = automap_base()
Base.prepare(engine, reflect=True, generate_relationship=_gen_relationship, name_for_scalar_relationship = name_for_scalar_relationship, classname_for_table=camelize_classname, name_for_collection_relationship=pluralize_collection)
Buyer = Base.classes.Buyer
Base.query = db_session.query_property()
I didn't show the naming functions because it's heavy and not important for the problem (i guess)
And here's my schema
import graphene
import json
from graphene import relay
from graphene_sqlalchemy import SQLAlchemyConnectionField, SQLAlchemyObjectType
from database import Buyer as BuyerModel
class Buyer(SQLAlchemyObjectType):
class Meta:
model = BuyerModel
interfaces = (relay.Node, )
class Query(graphene.ObjectType):
node = relay.Node.Field()
all_buyers = SQLAlchemyConnectionField(Buyer)
schema = graphene.Schema(query=Query)
Produces this error :
AssertionError: You need to pass a valid SQLAlchemy Model in Buyer.Meta, received "<class 'sqlalchemy.ext.automap.Buyer'>".
The problem could be linked to the fact that graphene updated to 2.0 but i could find any clue in the docs
gleaming frost
Anyone know a good link or website for editing jsons for things like "Phone List" or "player name and ID"
spice drift
if i'm running a discord bot that connects to a DB does it make sense to be opening and closing that connection per command?
or should I just be committing the changes as the bot edits stuff
tropic zealot
You should probably open and close the connection any time you wanna do something.
ornate abyss
I think for something like a discord bot, keeping the connection open for the entirety is a better option. Creating a connection to the database takes time, and you dont want to be opening and closing say every message.
It really depends on the amount of queries you are going to be making
neat arch
Is there anyone here who has experience with manipulating csv files in python?
vestal apex
I'm sure a number of people do! do you have a particular question about the topic?
neat arch
Oh, I moved it to the python subfolder. Last time I asked I did not receive a response and do I'm hoping I would get a response and I did
😊
ornate abyss
@neat arch what is the actual issue
stray zodiac
I have two tables and want to select some data
I only want to get the data from table 2 if a row in table 1 has a certain value
what's the best way to do this and is it possible in one query
hope that makes sense
neat arch
Kinda like that ^
But es fine i think I know to resolve it
Will let know if there are issues
ornate abyss
@stray zodiac use a join
stray zodiac
Could you perhaps explain a little more? I looked into joins but wasnt sure which one to use/how to go about it
ty for helping 😛
ornate abyss
Well, im assuming the tables are linked via some sort of foreign key?
stray zodiac
yeah they are
ornate abyss
So, can you provide the schema
so i can get a better idea of what you want to do
But from what i gather, you would do an inner join on where the id = foreign key AND table.column = "some val"
ornate abyss
np
stray zodiac
So I have two types of questions, multiple choice and single answer
I want it to only get the options from the Multiple_Choice table if Questions.style is multiple choice
ornate abyss
SELECT option1, option2, option3 FROM Multiple_Choice INNER JOIN Questions ON Multiple_Choice.question_id = Questions.id AND Questions.style = "multiple choice";```Assuming the text within style is "multiple choice"
ornate abyss
Np :)
stray zodiac
So here's what I have from your help
SELECT Questions.text,Questions.answer,Multiple_Choice.wrong_answer1,Multiple_Choice.wrong_answer2,Multiple_Choice.wrong_answer3 FROM Multiple_Choice INNER JOIN Questions ON Multiple_Choice.question_id = Questions.question_id AND Questions.style = multiple_choice ```the problem is that it only returns all the multiple choice questions, can i get into return short answer questions too
so basically if the question is multiple choice it returns 5 values (text, answer, option 1, option2 and option 3)
and if it's short answer it returns just 2 (text,answer)
ornate abyss
You would most likely have to make two queries
Since the columns you are wanting to select are different
ornate abyss
Glad I could help :)
zealous elk
Any good tutorials out there for learning good database structure practices?
I just finished the django tut and i'm working on my app. I realized i have no idea what kind of models i'll need XD
ionic pecan
well, think about how it would make the most sense
for example, what data do you plan on storing?
how does it relate to each other?
just whatever you do, try to make sure your queries dont look like this at some point ```Py
Title.objects.annotate(entries_last_week=Subquery(Entry.objects.filter(entry_date2__gte=make_aware(datetime.datetime.today() - datetime.timedelta(days=7)), entry_title=OuterRef('pk')).exclude(entry_readability=False).values('entry_title').annotate(cnt=Count('*')).values('cnt')[:1], output_field=IntegerField()),).filter(entries_last_week__gte=5).order_by("entry__entry_date2")
its a whole science for itself tbh. And as long as you dont have big performance requirements that good database schemes are not really a must have
ionic pecan
courtesy of @oak pelican
oak pelican
Again @ionic pecan
That is not my but django 1.11's fault
In django 2.0 it looks like
all_titles = Title.objects.annotate(
num_entries=Count(
"entry", filter = Q(entry__entry_date2__gte=make_aware(
datetime.datetime.today()-datetime.timedelta(days=7)
)
)
)
).order_by("num_entries")```shhhhhh
oak pelican
from django.urls import reverse
from django.http import HttpResponse, Http404, HttpResponseRedirect, JsonResponse, HttpResponseForbidden, HttpResponseNotFound, HttpResponseServerError, HttpResponseBadRequest
from django.core import exceptions
from django.template import loader
from django.contrib.auth.models import User
from django.contrib.auth import authenticate, login, logout
from django.core.exceptions import ValidationError
from django.core.validators import validate_email
from django.core.paginator import Paginator, PageNotAnInteger, EmptyPage
from django.db import IntegrityError
from django.conf import settings
from django.views.decorators.csrf import ensure_csrf_cookie
from django.core.mail import send_mail as original_mail
from django.utils.html import mark_safe
from django.db import models
from django.contrib.auth.models import User
from django.utils.timezone import make_aware
from django.core.mail import EmailMultiAlternatives
from django.db.models import OuterRef, Subquery, Count, IntegerField``` @broken lindenfuck me
oak pelican
Do these come built in
zealous elk
HOLY HELL
zealous elk
Is that real
broken linden
oh and someone
zealous elk
is that a real thing
oak pelican
Yeah it would take me about a year two implement them all myself @broken linden
zealous elk
That's bad right?
broken linden
nah
oak pelican
@zealous elk Nope
ionic pecan
django has pretty much everything built-in, it's amazing
oak pelican
Yeah
@zealous elk Those are prewritten amazing codes which you don't have to write if you use django
oak pelican
If you use flask you have to implement them all
zealous elk
Meh, whatever
broken linden
like the first three lines are stuff you dont even need to import in flask
everything else is there via the rich flask plugins ALSO this proofs my point right of django forcing you to do stuff the django way
@oak pelican
zealous elk
anyways, i'm going back to the start of my project because i need to learn how to do some TDD
but it's not really intuitive to me.
Does it ever become intuitive?
oak pelican
That doesn't force anything dude lol, you can import abstract classes of all of them and build over themselves, or you can even screw them and build anything freely
oak pelican
The only thing it forces is the base directory structure
oak pelican
pffffft
broken linden
(python noises)
zealous elk
XD
omg
I never realized there was a war between flask and django
why not just use both
oak pelican
how
broken linden
wolfturn
zealous elk
a nasty hyrbid
broken linden
both parties would kill you for that
zealous elk
hybrid*
oak pelican
lol
zealous elk
LOL
broken linden
its war
zealous elk
i'm gonna learn both eventually
oak pelican
You can learn flask
zealous elk
i hear flask is easier for quick and easy stuff
oak pelican
Because it is really simple
broken linden
why do you want to learn django first?
zealous elk
Because its the harder one
oak pelican
So I'd suggest learning flask if you are planning a small project
oak pelican
lol
zealous elk
this is the first time i've used a professional level framework like this.
broken linden
actually no django has more provided structures and clear ways todo shit
zealous elk
can i see?
zealous elk
I'd love to see what some actual views look like in django
broken linden
(on the other side its someone...)
zealous elk
because i'm not sure how to structure mine yet.
oak pelican
That would never happen in flask because flask can't even handle that kind of website xD
zealous elk
So general views are pretty much the standard for django no?
broken linden
11/10
oak pelican
wat
oak pelican
lol aight :D
broken linden
@oak pelican which website show me
oak pelican
Dude I'm just kidding :D
broken linden
but
zealous elk
as_view()
broken linden
its wartime 😭
zealous elk
That thing
oak pelican
But it is obvious that it is easier to make a heavier website in django
That is for class views Wolf
zealous elk
Yeh, class views
broken linden
(surpressing of long hmma)
oak pelican
If you want freedom you'd use function views
zealous elk
Function views. let me google that.
oak pelican
Classviews are pretty limited
@ensure_csrf_cookie
def index(request):
if request.user.is_authenticated():
styleconfig = UserProfile.objects.get(user_name=User.objects.get(username=request.user.get_username()))
title_count = styleconfig.user_title_pref
else:
styleconfig = None
title_count = 50
template = loader.get_template("mainsite/mainsite/index.html")
template2 = loader.get_template("mainsite/mainsite/index_logged_in.html")
context = {
"index_name": request.COOKIES.get("titles") if request.COOKIES.get("titles") in [*TITLECOOKIES.keys()]+[str(chn) for chn in TitleChannels.objects.all()] else "today",
"site_name": URL,
"all_titles": getlast(Title.objects.filter(title_channels__channel__contains=TitleChannels.objects.get(channel=request.COOKIES.get("titles"))).order_by("entry__entry_date2") if request.COOKIES.get("titles") in [str(chn) for chn in TitleChannels.objects.all()] else TITLECOOKIES.get(request.COOKIES.get("titles"), TITLECOOKIES["today"]), title_count),
"title_channels": TitleChannels.objects.all(),
"styleconfig": styleconfig,
"index_body": index_body,
"index_header": index_header,
"empty_title_list": empty_title_list
}
if request.user.is_authenticated():
returny = HttpResponse(template2.render(context, request))
returny.set_cookie("visited_index", True)
return returny
else:
returny = HttpResponse(template.render(context, request))
returny.set_cookie("visited_index", True)
return returny```Here is my index
Don't mind that one line
"all_titles": getlast(Title.objects.filter(title_channels__channel__contains=TitleChannels.objects.get(channel=request.COOKIES.get("titles"))).order_by("entry__entry_date2") if request.COOKIES.get("titles") in [str(chn) for chn in TitleChannels.objects.all()] else TITLECOOKIES.get(request.COOKIES.get("titles"), TITLECOOKIES["today"]), title_count),```oh gosh
oak pelican
Nice line isn't it
zealous elk
Why are there 2 templates O.o
broken linden
(you will be happy if you know)
oak pelican
broken linden
those functions are called "god functions"
oak pelican
See you in 10
zealous elk
Meh, i already built it
Just kinda threw it in there
if it doesn't work i'll go back out and start it over
broken linden
if you use a system which makes sense to you and dont need high avaiability and performance stuff dont change it
zealous elk
eventually
WEll
this is a project i want to be able to put on my resume
so i'm trying to make it the nicest i can.
broken linden
¯_(ツ)_/¯
zealous elk
so its like XD
but since i don't know what i don't know.
I might as well just do it quick and dirty and then find out what i did wrong i guess
broken linden
i put the majority of shit on my github and in case i ever open a resumee ill prolly just generate a page out of all my github projects automatically regardless of them being shit or not
oak pelican
I'm gonna head out but before that I have to say this @zealous elk
broken linden
if my github stuff is toooo shitty i delete it
oak pelican
Whatever you do
broken linden
dont use django
oak pelican
Don't put your first project on a framework on your resume
I made this website as my first project
broken linden
this whole resume thingy is so funny for me as a german tbh
oak pelican
It has like 4k lines of code
broken linden
a resume does nearly not matter in germany from my experience
at least in programming
oak pelican
Yet I could've made it 10x more efficient with my knowledge now
gusty spindle
🙆 ping
broken linden
aaaah
zealous elk
@oak pelican Thanks, then i'll quick and dirty it and then use that to re-make it
i'll do test driven development later then.
oak pelican
I mean there are a lot of parts of a website
The messaging is a whole different part if you are gonna do a social media
The message app could take a lot of time alone
zealous elk
no messaging
oak pelican
API is another bitch
zealous elk
It's just a simple quiz + information generation i guess is what i could call it?
depending on your inputs
and saving a user.
not a lot
broken linden
i mean @zealous elk look at it https://github.com/hargoniX?tab=repositories 2 of them are jokes, one is just editing a single word in a readme i didnt contrib to one even soo ¯_(ツ)_/¯
oak pelican
Your models are also another thing, you should really build your models neat if you are gonna have more than 4-5 models
My models suck in my website
zealous elk
ok
Yeah i'm kind of OCD
so that's why i wanted to learn models really well
so i could make them super neat.
oak pelican
Actually I'd share my website with you if I hadn't stole the html and css and half of the javascript
lol
zealous elk
lol
broken linden
(he is lying there is no website)
zealous elk
XD
Thank you all
you guys are really motivating 😃
i'm switching from real estate to coding and its a really hard thing to switch to cold turkey so i'm glad you guys are around
i hope i can eventually make some money from this lol
been studying non-stop for about 3 weeks now.
zealous elk
i hope i can be good in 3 months.
oak pelican
I hope it works out for you :D
zealous elk
😃 thanks again!
rain karma
ogod, that looks ugly, one sec
Projects.ProjectName as 'Project Name',
EmployeeProjectAssignments.StartDate as 'Start',
EmployeeProjectAssignments.EndDate as 'End',
(Employees.FirstName + ' ' + Employees."Last Name" + ' ' + CAST('Start' AS VARCHAR(25)) + ' ' + CAST('End' AS VARCHAR(25))) as 'Manager'```That looks a bit better
how do i cast these two Start and End to work?
hexed mango
Anyone know how to retireve every nth document from a collection in mongodb? I want to process this on the mongo server not on my backend server
shut crater
Is there any way to insert columns into a query statement by column number in a table instead of column name?
Like I'm trying to do here, by using col[1] col[2]:
c.execute('INSERT INTO {} (Date, {}, {}, Source, Headline) VALUES (?, ?, ?, ?, ?)'.format(table), (Date, col[1], col[2], Source, Headline))
No
shut crater
Alright, that's what it seemed like on the internet too. No worries, thanks for the confirmation
ornate abyss
np :)
tawny sail
hey guys
broken linden
👋
tawny sail
im new to flask and im figuring out a way to connect to mysql
need help plz
mysql = MySQL()
# MySQL configurations
app.config['MYSQL_DATABASE_USER'] = 'jay'
app.config['MYSQL_DATABASE_PASSWORD'] = 'jay'
app.config['MYSQL_DATABASE_DB'] = 'BucketList'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
mysql.init_app(app)
is it something like this?
should it come before
app=Flask(__name__)
or after ?
broken linden
considering you need the variable "app" for it it would kinda make sense to have the app variable defined before using it
tawny sail
ok
frozen cairn
Hi guys! Does anyone have a few minutes to help me out? I need to use a flexible way of connecting to a certain host ( postgress, SQLite, MySQL etc. ) and create tables, and populate them.
I think a combination of a package called Munch ( Turning a list of dicts to objects ) and SQLAlchemy but i cant seem to get it to even connect.. Been using python for little over 2 months now so still new.
Any help would be greatly appreciated!
frozen cairn
Well i learned that the SQLAlchemy needs to have a class or objects to parse and insert those objects into the database. But it needs to be generic ( i dont know what will be parsed, so tables must be made on the fly ) And munch is a tool that can create an object from a dict. But then again im new so i might be wrong.
frozen cairn
So it can take a set of dictionaries, match them to a table and insert them too? Thats amazing. Thank you. Ill put some more time into it. Is there any other package that i might need to achieve this?
ionic pecan
i don't understand what you mean with take a set of dictionaries
do you want to create tables from dictionaries?
frozen cairn
I have a list, which contains dictionaries. And i need to find a way to connect to a database ( will be MySQL DB most of the time ) .. And some how create tables, corresponding to the keys of those dictionaries
And then populate the tables with the values of those dictionaries.
frozen cairn
Well since im rather new, i dont know how to create classes on the fly ( which i thought you needed to create those objects, which Alchemy can use to store in the DB )
viral crag
You don't have to make the classes on the fly
The idea is what you write the classes yourself
They provide a model which allows you to basically define what will be in your tables
Those classes are mapped to tables and instances of those classes are rows
You'll probably want something like Alembic as well
frozen cairn
I understand that, although i dont know what Alembic is yet. The list ( containing those dictionaries ) are completely different everytime. And i dont know how to generate a class, containing those attributes which i can use as a model.
viral crag
Well I don't understand why you would ever want to define your schema that way
But
You could probably use the lower level query building API
Or just literally generate the python code for the objects in like a string
ionic pecan
dynamically generating tables sounds like a bad idea, imho
what are you trying to accomplish?
frozen cairn
Well the idea is that a dataset is given, which can either be a two dimensional list ( or more dimensions ) , or a list containing alot of dictionaries which can be similar or different. The size can differ to from just a few dictionaries to hundreds or more.
When the set is given, it somehow needs to end up in a database. So credentials must be passed, and the dataset.
Then tables must be generated, and populated with the data inside those dictionaries or list.
I'm relatively new to python and i just found the alchemy package today. But i'm pretty stuck at this point.
viral crag
Honestly this system seems too generic to be useful
frozen cairn
Do you have any other suggestions?