databases

[manic tide]

which modern ones I will try them? i mainly use html css js py atm, thought about rust and c/assembly but i dont really make anything on those languages atm. I'm focused on mainly js and py right now for front-end development

js is my go to and py for backend, i just wanted to bring some more support for obscure language for niche projects

stuff people don't use

commonly

actually working on my own spaghetti syntax for html to basically make .py run like a .html document with a transpiler custom written but thats beyond me atm lol just another concept

[tardy latch]

got it, you might want to take a look at dbzero, we've just released it couple of days ago: https://github.com/dbzero-software/dbzero

[manic tide]

ok cool i just forked it

[tardy latch]

it's embedded and based on very similar principles as MUMPS, but built on a modern stack

[manic tide]

ive had this obscure angle the past year at developing software, all front end html based. I realized that AI can scale any html front end page into backend files easy, but getting that perfect spot on UI with all the buttons/links/spacing done perfectly that's more of an art form. back end is not really my specialty but I have built a few MCP servers for fun (until they started hitting my rate limits with automation on my free netlify lol) i had to take them down. basically mypoint is, for me by myself its just easier to focus on front end for now.

thats a cool project though I like the idea

i was running docker with yaml and pytorch trying to do quantum simulations but my laptop is not good enough to do anything cool, it was just a binary server trying to act like it was working in qubits

qiskit for quantum logic

i wanted to make a sort of "quantum model context protocol" it might work but im going to need at least a proper GPU x4 to test my theory

I like your snapshots idea in dbzero I need to add more stuff like that to my mumps idea

it was made to run in skulpt which renders py in html,,(webxos.netlify.app/injector made an app for that) which might not support "person" but i could make a work around

looks like i can i need to add that for user profiles etc snapshots

[coral wasp]

I'm still getting my head around dbzero... I'd imagine you run into nesting problems, what if A->B->C, and A is memo-ized? Is it transitive?

[keen minnow]

looks like a memory grid?

[compact fulcrum]

As for python classes in dbzero objects you can store only other memo objects. We got support for python colections, and types. If you want to store A->B->C all of A, B and C needs to be memo classes

[brisk narwhal]

does a unique index in sql databases prevent multiple identical insertions even for race conditions?

[thorny anchor]

yes

[tardy latch]

In dbzero you just work as with a regular Python with the difference that you need to add the @memo decorator to classes you want persistence for. This is it. For memo classes you get a ton of additional features not regularly available in Python such as: permanent UUIDs, tagging, composable search queries, aggregation queries, transactions, atomic updates and more - all the features you'd expect of a database but natively in Python and with 10x - 100x better performance. And it's not just in-memory, you can work with many TBs of data with limited RAM (NVME disks is recommended though). (see https://docs.dbzero.io)

[tardy latch]

It's not a memory grid. It emulates infinite memory by swapping to disk / network storage but with all transactional guarantees and intelligent caching.

[keen minnow]

memory grids do that too?

[tardy latch]

Well, maybe we can call it memory grid then - but it also extends Python with features for application development so it's more comprehsive.

[keen minnow]

any memory grid like infinispan or apache ignite can do that. And if it is just about caching memory to disk, then most caching library can do that (ex: caffeine/ehcache/etc.)

[keen minnow]

so the difference is the python annotation then?

(not saying it's bad or good, just trying to understand how it differs)

[tardy latch]

Right, it has very similar propreties but natively for Python (it's embedded)

[tardy latch]

memory grids, while powerful in my opinion offer poor developer experience. When implementing dbzero we started from developer experience first - we say that as a developer you should mostly focus on the logic and proper problem modeling (in terms of abstractions and data structures) - allowing scalability and availability to be addressed later (most of the projects may never even reach the scale to even bother about it). The core philosophy is - since most of the time when dealing with data we need to fetch them into memory (from database), why not let the data just stay in the memory forever ? dbzero allows you to build programs which never run out of memory.

[keen minnow]

Sure, nice!
How do you deal with data migration? Let's say I change the type of a field or add or remove a field?

And what happens if I haven't updated all my clients yet?

[tardy latch]

this can be addressed in multiple ways: 1) just adding a field to a dataclass definition or constructor acts similarly as adding a nullable column to a database (so when accessing old objects you will not get an error but None value which of course can be updated), 2) migrate decorators - if #1 (None as default) is not sufficient you can add "migration" methods to your objects, executed when an object is accessed with a new codebase (it does not migrate everthing, just the objects that you access), 3) instead of migrations oftentimes better approach is inheritance - just implement a derived class, 4) If a "heavy" migration is really needed (last resort) then you can start a separate process - which performs your custom migrate logic on live data but writing to a separate prefix, when done you can swap to new version in sub-second (depending of the data size). In situations 1 - 3 clients (other processes running old codebase) are not affectected (since your old code is still compatible with the object layout, unless there are some fundamental changes to the logic in which case the redeployment will be necessary)

[keen minnow]

Thanks for the reply!
Interesting!
I did hit you with tough questions, so I do appreciate the thoughtful answer

[tardy latch]

thanks :-). Your tough questions are appreciated

[keen minnow]

last question: do you provide any testing facility?
For instance, let's say I have a specific schema in use right now, and want to add some tests for a future change in schema and want to make it safe. What options would I have?

Goal being that I can add/change/remove a field and feel safe about it before it does hit the datastore

[tardy latch]

So, first of all - testing (unit, integration) should be done on dbzero right away (no database layer mocking), it's lightweight and does not affect performance of tests. While we don't offer any special framework for your particular use case I think it can be quite easily achieved in test suite by executing python process with multiple code-bases (currently in the project tests we just added scripts which simulate this). It's interesting problem though and I hope that when technology matures the community will implement such additional tools and frameworks.

[keen minnow]

Thanks! Worth a look

[tardy latch]

One more thing. In dbzero oftentimes migrations can be avoided by avoiding tight coupling. The tagging functionality allows you to use objects (or enums and even types) as tags which is an equivalent of a "relationship" in database. But in dbzero the mechanism allows you to relate instances which know nothing about each other and share no code (loose coupling). This might be a foundadion for truly composable architectures

[ripe scroll]

Is anyone aware of a good database schema builder application online?

[cedar tiger]

I really like https://dbdiagram.io/home

[ripe scroll]

Thank you I found it! I was looking for this site everywhere and just didnt remember the name 😄

[tardy latch]

Hi Guys, sharing another dbzero example. Time-travel your Python's memory with the db0.snaphsot function:

[cedar tiger]

<@&831776746206265384> scam

[brave bluff]

I'm having some trouble with SQLAlchemy

class PM(MappedAsDataclass, AsyncAttrs, LegacyBase):
    __tablename__ = "pms"
    __allow_unmapped__ = True

    history_ids: Mapped[List[int]] = mapped_column(
        "history", JSON(), default_factory=list
    )

    _history: list["PM"] = field(init=False, default_factory=list)

    async def get_history(self):
        if not self._history:
            session = object_session(self)
            if not session:
                raise Exception("No session")
            self._history = list(
                session.scalars(
                    select(PM)
                    .where(PM.id.in_(self.history_ids))
                    .order_by(PM.datestamp.asc())
                    .options(joinedload(PM.recipient), joinedload(PM.sender))
                )
            )

        return self._history

With this code, I'm getting AttributeError: 'PM' object has no attribute '_history'. Did you mean: 'get_history'? and I don't know why. I tried adding __allow_unmapped__ = True and also adding repr=False to the field.

[topaz iris]

Hello guy I need expert Python developer for Capcut APK device register
I'm willing to pay a reasonable payment after you complete my work

[oak swallow]

Okay

Nice to meet you!

I can help you.

I am a senior AI and full-stack developer.

Please explain about your idea in more detail.

[topaz iris]

Inbox me bro

[oak swallow]

I have sent you request

[crisp gorge]

SQLAlchemy skips dataclass initialization on DB load, so _history doesn’t exist; initialize it manually or via @reconstructor.

[grim vault]

!rule 9

[delicate field]

Rules

9. Do not offer or ask for paid work of any kind.

[hollow abyss]

I need help ASAP.

With my database.

@crisp gorge

[delicate egret]

is this project legit?

https://pypi.org/project/synadb/

[crisp gorge]

Ask

[coral wasp]

Its first release is a month ago. I wouldn't use it for anything other than curiosity.

[west bloom]

im planning on creating a small database around 15 items, would sqlite be suitable or are there better options.

[dawn shard]

SQLite is fine

[hollow abyss]

yo sam

@crisp gorge

Help man.

[hollow garnet]

All you have said is "Help". No one here is a mind reader that somehow magically knows what your issue is. You have not told us what you are having an issue with specifically. You haven't given us any context to help whatsoever.

[river belfry]

not related to database , please go id:customize to see what suites this text for you

[earnest glen]

What is the overlap between Polars, Pandas, Apache Spark, and SQL? I keep seeing both Polars and Pandas talking about joins, aggregates, and other big data (I'd guess) terms and gesturing at SQL (for example, you can issue SQL statements in Polars to do the same thing, I saw a lot of overlap in the terminology used in Apache Spark (which is used for SQL data analysis) and Polars/Pandas, and they (Polars/Pandas) can export to SQL for some reason. I am trying to figure out how to get a list of every unique row in this CSV I have (which is currently my standin for a DB until I get an actual one setup) and am having trouble figuring out how I should put together the miriad functions in either Polars or Pandas to do it

Also, not entirely sure if this is a DB specific question or a Data Science one, though technically I am doing a little bit of both right now so I suppose both make sense

[earnest glen]

I did some research, it seems people genuinely use SQL for data analysis, and that the exact thing I wanted to do was a big part of the SELECT and JOIN snytaxes that I hadn't really learned anything about yet

[slate sinew]

Hi, yessfan. I see you're having a hard time in grasping some data/computing ideas and vocabulary, let me try to help you.

Short answer:

• SQL is a language to "talk" to databases (DBMS) or some data processing libraries (like pandas or polars). But there are other ways to "talk" to those software/libraries, like using their API through method calls.
• To identify unique rows in a csv file, you can use a data processing library like pandas or a DBMS, but pandas is more suitable for your case.
• The overlap between concepts will exist because they are about the same domain (data processing). Its like learning different human languages to talk about the same subject, you'll find similar words and expressions, and there will be a lot of overlap.

Long answer:
First things first, the main idea when it comes to use computers and software is that it works like this:

1. you provide some data as input
2. the computer/software will process it
3. it will return the result to you

That said, in your case, you have a data source and want to identify the unique rows (or records). The first thing that you will need is some engine/software to process it and there are several. Here are a couple of them:

1. A DBMS (Database Management System) like postgres, mysql, mongodb, etc. Pointing out some things about these pieces of software:

   • Those are usually used for more complex contexts to store, manipulate and visualize data. But I assume your scenario is not that complex since you have just one csv file. So using this solution would be overkill.
   • To "tell" the DBMS how to process and present the data you want, we use some language, SQL for instance. SQL is usually used for structured data (on postgres and mysql). So SQL is just a language to describe what you want, but you need a software to interpret it and process the data based on your SQL sentence. The software to interpret it could be a DBMS as mentioned or, leading to our next example, a lib like pandas or polars.

2. Using a lib (pandas/polars):

   • This is a more suitable solution in your case.
   • The way we "tell" those libs to process the data, we want could be using SQL since some of them provide the feature of interpret that language, but the most usual way to do this is through method calls provided as the lib's API.

Now let me give you an example of how you could identify unique rows on your csv file.
First of all, you need to define what makes a row unique among your data. Let's suppose your rows are cars and you have the following information about them: license plate, model, color and manufacturer. Lots of cars will share model, color and manufacturer, so those can't be used to identify unique cars, but license plate is a unique information per car (or row) in this example. And the code would be something like:

import pandas as pd

# Load the csv file into a DataFrame
df = pd.read_csv('cars.csv')

# Identify unique rows based on the 'license_plate' column
# the `subset` arg here will receive the list of columns that define uniqueness
unique_cars = df.drop_duplicates(subset=['license_plate'])

# Save the unique rows to a new csv file
unique_cars.to_csv('unique_cars.csv', index=False)

P.S.: Apache Spark is another data processing engine that could be used for this task, but it is more suitable for big data scenarios and distributed computing. For a single csv file, pandas or polars would be more efficient and easier to use.
P.P.S.: The exportation to sql from pandas or polars is just a way to save your dataframe into a database. It is not necessary for your case, but if you want to do it, you can use the to_sql method from pandas or polars.

[earnest glen]

I figured it out already im good

[strange stratus]

is an ORM always preferred over raw sql or query builders?

[storm mauve]

not necessarily
sometimes ORMs will lack support for features you need (specially new or non-standard things), and in some cases raw sql can be more performant than using an ORM (whenever or not that matters depends on your use case)

you should almost always prefer using a query builder than raw sql when it's viable though

[strange stratus]

Ok, I'm just starting out with databases and python and I'm trying to learn SQLAlchemy but I have also heard others recommend Tortoise and other ORMs though I'm just wondering if it's really a big difference and matters, I'm guessing the only difference here would be how to interact with the libraries and its features I guess?

[dawn shard]

For the vast majority of projects (especially generic CRUD stuff), you should probably use an ORM, and the specific ORM won't make a difference

[strange stratus]

Ok, thanks!

[frail ember]

hi

[thin lotus]

should I go with asyncpg or psycopg 3?

asyncpg seems better in a smaller scale from benchmarks

https://fernandoarteaga.dev/blog/psycopg-vs-asyncpg/

[empty nebula]

@rough hearth ban someone promoting

[rough hearth]

!warn @steady shard your message was removed for advertising or soliciting survey participants, which are not allowed.

[delicate field]

:incoming_envelope: :ok_hand: applied warning to @steady shard.

[empty nebula]

Gud

[rough hearth]

it's not "good". I'm just letting people know what the rules are so that they can engage with our server properly.

[empty nebula]

Nah you arr doing good

[split rapids]

Hi

[sinful otter]

I just finished my crud operation using fastapi and I want to learn databases.can anyone give advice where to start or are there any tutorial available for free on YouTube

[rugged zephyr]

I was wondering if any of you fine people could offer some advice to an amateur developer about database hosting for a small-scale POS application. The app includes a real-time search bar that queries the database as the user types, so I’m trying to find a solution where those queries don’t lag to the point of affecting usability.

[versed shore]

Are you asking about a database host that would be fast enough to do that, or are you asking about how to build the application so the search bar doesn't lag?

[rugged zephyr]

database hosting

[rugged zephyr]

ideally I would prefer to build it better but thats a later problem

[latent yarrow]

honestly, any db with an API can do this with an easy javascript function

[rapid horizon]

I would assume your best bet here would to find a good VPS/VDS to host your database? OVH, Aruba Cloud, Digital Ocean, Vultr, Bitlaunch, etc.. You can setup MySQL (or whatever engine you prefer) and perform your operations there. Are you speaking in terms of as you're building this, or do you have a different solution currently in place?

[rapid horizon]

Google any MySQL tutorials, it will get you familiarized with databases, queries, stored procedures, transactions, etc, CRUD operations can actually be intensive if you're enforcing strict scrutiny

[cedar tiger]

Could look at sqlite, so that the POS can work offline and the read and write abilities are on the local database first, then push changes to the cloud at an interval.

[versed shore]

I'm quite sure that this type of UX design is usually also implemented using a worker that doesn't block/lag the UI to query the database. And preferably also a fast database, of course.

[severe mesa]

anyone wants to clean their data or wants to find insight of your data or want to apply machine learning i will be you data analyst

[barren abyss]

Is current pipeline ETL/ET in py fast enough?

[quiet nebula]

Is there any async sqlite3 database that suppports zstandard compression?

There is aiosqlite without compress, and there is a lib that uses rust for database compression

[gaunt meteor]

try libsql

but last time ichecked sql lite nativly runs byte compression anyway

[analog rose]

does any one can teach me API?

[cloud cape]

Hello friends!

What is your favorite time series database?

[ionic pecan]

rrdtool

[mint canopy]

Can sqlite hold up to 100 entries a day or maybe more?

[thorny anchor]

sqlite can probably get to tens of billions a day

[brave bluff]

I have a question in the ever eternal debate of ORM vs raw queries. A recent thought has been about how one advantage to any decent ORM is when you do an insert/update, it should only be writing the query to include the fields that are set/updated, meaning a more efficient query, in contrast to writing a query that contains every field, even if the field isn't used/has a default/isn't updated. The question I'm trying to determine is if that efficiency is worth using an ORM. I could create models via dataclasses (which is what SQL Alchemy does anyway, the ORM I'm currently using), but then if I want to save an updated object, I'd need to loop through every field and save them all, or write some logic that on setting/updating a field, it saves an internal tracker of those fields, which are then used to update, thus increasing the logic and taking steps towards an ORM anyway.

I'd love if anyone has any thoughts to this.

[limpid owl]

hi

[icy glacier]

If you do raw queries you're going to be updating every single field on every update of your models, or implementing change tracking yourself, and you'll probably be doing it for every model or finding a pattern and making a library to do it for you, as you've pointed out. I think that it is worth it to just use an ORM, I personally don't want to spend my time managing that logic in any capacity and would find it less productive and quick to get something operational.

I think the side of reading from the database using your orm for presenting information is a more interesting question, such as how much data is being fetched in that isn't needed for it? Or how much data to you need to fetch each time because of eager loading stuff you have in place for when you're writing, or the opposite, how many round trips do you need to make because of lazy loading for writes?.

[brave bluff]

Thanks for the feedback. I'm using SQLA, and I try to be explicit about fields when I can, and only eager load when I NEED the data. So I guess it does come around, as you said, to just not wanting to write the insert/update logic, which the ORM handles. I often end up writing custom queries anyway, and just using the returned dicts, but when I do need the full model, the built in logic of SQLA is often useful.

[tidal raven]

Hey guys, hope you all are doing great.

I'm a Business Analyst and I'm looking for some public API related to Financial, Predictive Analysis. I already did something using the PokeAPI and the RaMAPI.

If someone know any good API please, let me know, also if there's a better channel to talk about it.

[leaden marlin]

Hey guys, im a professional data systems engineer and im just looking for some friends in the field, add me!

[cedar tiger]

what is a data systems engineer?

[leaden marlin]

Essentially just a data engineer except I work on multiple different systems new and old

[uncut void]

Hi folks,
Using Flask+sqlalchemy (MySql) with celery
How to setup the database connection for celery tasks ?

[analog ravine]

hey i m using sqlalchemy
how can i alert/modify tables to include relationship and some other columns without deleting tables ....

[torpid cosmos]

Hi I need to use python for mathematical graphs and I have never used python before as well I have no programming idea for graphs what should I do next

[versed shore]

You should probably look into Alembic. It's a tool made by the same organization as SQLAlchemy. It's used to generate scripts to migrate your database when you make changes to your schema via SQLAlchemy.

[versed shore]

I think you're in the wrong channel. Might have more success in #data-science-and-ml . To get you started, look into the matplotlib and numpy libraries, they're the industry standard for everything with math and plotting graphs

[fair cloud]

dbt-core / Trino: "Access Denied" for catalog not used by selected models

I’m running dbt with some selected models by tag. However this fails with a TrinoError saying I don’t have access to certain catalogs not used by the models I’m trying to run.

It looks like dbt trying to access the gold catalog when I’m only running bronze models that have no refs to gold? Does dbt validate/check access to all configured catalogs during parsing or compilation, even for models outside the selection?

For reference, we have a setup similar to this:

snapshots:
  my_project:
    +database: "{{ 'sandbox' if 'local_dev' in target.name else 'silver' }}"
models:
  my_project:
    bronze:
      +database: “{{ ‘sandbox’ if ‘local_dev’ in target.name else ‘bronze’ }}”
    silver:
      +database: “{{ ‘sandbox’ if ‘local_dev’ in target.name else ‘silver’ }}”
    gold:
      +database: “{{ ‘sandbox’ if ‘local_dev’ in target.name else ‘gold’ }}”

Here is what I’ve tried:

dbt run -s tag:my_tag
``` which results in

TrinoUserError(type=USER_ERROR, name=PERMISSION_DENIED, message="Access Denied: Cannot access catalog silver", query_id=...)

I then tried excluding the snapshot

dbt run -s tag:my_tag ––exclude resource_type:snapshot

TrinoUserError(type=USER_ERROR, name=PERMISSION_DENIED, message="Access Denied: Cannot access catalog gold", query_id=...)

[mild agate]

im looking at a explaination to MongoDB for python and have a question, how would i use a variable as the input? (eg "name": "[variable]")

[dawn shard]

Would "name": variable work? Assuming variable is a string.

Well, you shouldn't assume that, but instead check to prevent injections.

[sick ginkgo]

Hello , does anyone need a backend developer?

[minor venture]

I am making a project and the team wants to use an ORM. I would likely use SQLAlchemy as I have heard it works well with FastAPI. The main thing I am looking for is the ability to treat returned rows as objects for type checking/name validation purposes. I am not super keen on creating the schema from the code. In the past I have just used raw sql with asyncpg. After taking a quick look at SQLAlchemy it seems to add a lot of complexity as you are required to essentially define the entire schema in the code. Would I have better luck making my own partial ORM with Pydantic (dont have much experience with this either) where I convert the returned SQL into objects specific to function? IE User object from get_user function.

[versed shore]

If you want very barebones: sqlite supports the row factory pattern. You can pass a callable to a cursor, which will get called for each row. This function could then construct an instance of your domain object class each time you fetch a row from the cursor

[minor venture]

do you know if asnycpg supports that

Looks like this might be what I want
https://magicstack.github.io/asyncpg/current/faq.html#can-i-use-dot-notation-with-asyncpg-record-it-looks-cleaner

Would you say that not create the schema in the code is a good enough reason to not use an ORM

Part of the problem is the schema already exists, so I dont want to have to copy over its entire structure

[minor venture]

Well, I did find a way to generate SQLAlchemy code via sqlacodegen made by a name I recognize from here

[craggy arrow]

Hey, if I wrote a database related library, and wanted some feed back on it, would here be an okay place to link it? Or somewhere else, I don't want to be rude / spam the wrong channel. Edit: I'm also aware that space is crowded 😅

[versed shore]

Looks like it yeah. I'm sure it will work for type checking, at least. I'm looking at their implementation of Record but it's a C extension so I'm out of my depth there. I can't find the constructor signature, but if you can subclass Record and transform the k/v form (as it is in Record) into an object with attributes, you basically have the same functionality of the row factory from SQLite I mentioned before. Worst case, you just iterate over self.items() and dump it into attributes via setattr on the fly in __init__ and blanket the signature via *args **kwargs.

[craggy arrow]

Does my library maybe smooth out what your recommending here? I haven't added asyncpg yet, but plan to:
https://github.com/jimcarreer/dinao
Basically you can decorate functions, and then it uses the type hints to figure out automatic mapping to templated SQL as well as automatic mapping to result sets. I specifically started this library back in 2020, because I actually don't like ORMs much.

I just release 2.1 btw, which added the asyc support. What I will say Ice: you probably will hit bugs, but there's an example of fastapi + async with this using postgres.

[minor venture]

Your library actually looks really interesting. I might actually use it in my personal projects once you add asyncpg support. However the project I am currently working on needs to have a more mature and widely used library.

[craggy arrow]

Yeah no hard feelings over that, I wouldn't even push it on my coworkers even thouogh we bypss the ORM in SQLA all the time and just write plain sql

I'll priortize asdding support for asyncpg; I think the only tricky bit is the templating symbol is dynamic, but that shouldn't be a big deal honestly.

[minor venture]

Just a question, will the execute and query decorators always be over a function with no body? Or would you sometimes add something to the body?

[craggy arrow]

Think Ice was talking about my specific library

[craggy arrow]

Yeah generally the body's empty, its basically "this function maps to this query" But you can run multiple "bound" functions within a single transation if you need it, to do more complex things:

@binder.transaction()
async def populate(cnx: AsyncConnection = None):
    await make_table()
    await cnx.commit()
    await upsert(MyModel("testing", 52))
    await upsert(MyModel("test", 39))
    await upsert(MyModel("other_thing", 20))

It also gives you access to the connection object.

For commits, etc ...

[minor venture]

So each execute/query function is intended to be a 1-1 mapping of a SQL statement

[craggy arrow]

Yeah basically

I want to MAYBE implement for / if / etc in the templating engine

but Im not sure its a good idea.

[minor venture]

Yeah, I will check this out next time I am working on my discord bot project because currently my database operations are just through
⁨```py
async def execute(self, sql: str, *args) -> None:
conn = await self._acquire()
await conn.execute(sql, *args)
await self._recycle(conn)
async def fetch(self, sql: str, *args) -> list[asyncpg.Record]:
conn = await self._acquire()
rows: list[asyncpg.Record] = await conn.fetch(sql, *args)
await self._recycle(conn)
return rows or []

[craggy arrow]

Right! exactly, its basically just to reduce boiler plate, Ill try and get asyncpng in next weekend.

[minor venture]

No need to rush just because of me btw that project has been on hold for a while so a couple extra weeks wont matter

[craggy arrow]

ha, well I mean full disclosure, I only have time for this now because I use claude heavily. Work started paying for it and is ... uh ... suggesting heavily we use it but before I risk a damn work incident I've been trying to use it on a personal account / projects to get a better idea of where the sharp edges are. So far Im pleased, though I really needed to add a MD specifically for it to get good results from this repo. I rejected like 3 implimentations it did for async before I got to one I liked.

[versed shore]

I just checked it out, nice work! Do you have support for rows affected? I find myself using that feature of cursors quite a lot

[craggy arrow]

Ah no, that is probably a good idea, would it be for something doing like updates/inserts generally? I've thought about mapping that on @execute(...) right now

Actually maybe I did do that already .... let me check

Sorry I wrote the core library like ... 4 years ago :D so some things may be out of my memory now

[versed shore]

mostly sanity checking. if my update statement affected more rows than it should, something is wrong and we need to roll back!

the upsert in your example does return an int

maybe that's it?

[craggy arrow]

Yeah I just want to be sure that's what it is but pretty sure that's correct.

Yup:

    async def execute(self, sql: str, params: tuple = None, commit: bool = None) -> int:
        """Execute the given SQL as a statement with the given parameters and return the affected row count.

        :param sql: the SQL statement(s) to execute
        :param params: the values to bind to the execution of the given SQL
        :param commit: commit the changes to the database after execution, defaults to value given in constructor
        """
        commit = commit if commit is not None else self._auto_commit
        cursor = self._cnx.cursor()
        await self._execute(cursor, sql, params)
        affected = cursor.rowcount
        if commit:
            await self.commit()
        await cursor.close()
        return affected

[versed shore]

cool

[craggy arrow]

Anyway, like I said, its not been battle tested anywhere, but I did write code to smack the crap out of a very basic example in flask and fast api, using multiple threads:
https://github.com/jimcarreer/dinao/tree/main/examples/fastapi

The templates are compiled / parsed once upfront, but aren't during when queries are run; so you wont take a hit from pyparsing there.

I've got a show case thread if y'all have more thoughts and questions:
#1470165258004860948 message
So I don't fill up this room talking about it.

[craggy arrow]

@minor venture couldnt help myself: https://github.com/jimcarreer/dinao/tree/feature/asyncpg-backend it'll be in the preview build for 2.2 here in a bit. I changed the fast API example to use it instead of psycopg v3
https://github.com/jimcarreer/dinao/pull/91/changes#diff-37626ac95f86fc0fc736f627634f32da35aa87a031c1d7013bc74808b46effe6

[fossil talon]

Hey guys,

How do you like an idea to make 1 general code for turning excel file into database?

Probably thats not something original lol

I guess I might need to implement the whole system of restrictions and recommendations, while choosing datatypes (and so on)

[blissful dove]

Hello, is there a database for user drawn numbers and letters? handwritten

[craggy arrow]

Do you mean like, a data set or a database that is specialized to store handwritten text?

[stoic apex]

hi im looking to get my sql funadamentals tight

are there any resources?

[versed shore]

Do you need this for yourself? Or is it just a general idea for a project? Because I can imagine this would be rather easy to implement for a given excel file, but very complicated to generalize to 'all' excel files

[wanton hemlock]

Okay

[fair cloud]

Sorry, reposting this here in hopes that someone could help

dbt-core / Trino: "Access Denied" for catalog not used by selected models

I’m running dbt with some selected models by tag. However this fails with a TrinoError saying I don’t have access to certain catalogs not used by the models I’m trying to run.

It looks like dbt trying to access the gold catalog when I’m only running bronze models that have no refs to gold? Does dbt validate/check access to all configured catalogs during parsing or compilation, even for models outside the selection?

For reference, we have a setup similar to this:

snapshots:
  my_project:
    +database: "{{ 'sandbox' if 'local_dev' in target.name else 'silver' }}"
models:
  my_project:
    bronze:
      +database: “{{ ‘sandbox’ if ‘local_dev’ in target.name else ‘bronze’ }}”
    silver:
      +database: “{{ ‘sandbox’ if ‘local_dev’ in target.name else ‘silver’ }}”
    gold:
      +database: “{{ ‘sandbox’ if ‘local_dev’ in target.name else ‘gold’ }}”

Here is what I’ve tried:

dbt run -s tag:my_tag
``` which results in

TrinoUserError(type=USER_ERROR, name=PERMISSION_DENIED, message="Access Denied: Cannot access catalog silver", query_id=...)

I then tried excluding the snapshot

dbt run -s tag:my_tag ––exclude resource_type:snapshot

TrinoUserError(type=USER_ERROR, name=PERMISSION_DENIED, message="Access Denied: Cannot access catalog gold", query_id=...)

[fossil talon]

The key of my idea is to generalize😁And yes, thats for myself😊

[fossil talon]

The general idea is to put a set of constraints first, and removing them step by step while further updatings, as that usually works in agile models

I mean thats my own project😁anyone, who wish to join me for project, please, contact through dm

[versed shore]

not a bad approach. i'm currently swamped with my own project, good luck though, and i'd be interested to see your results!

[fossil talon]

Thanks man

[mild agate]

im trying to make a discord bot that saves stuff to a database and want to know how i would get a entry split into different variables

i currently have
idget = currency.find({"id": id})
("id" being the discord ID searched)

if what i want is the second index would it just be
amount = idget[X] (with X being the index)

[mild agate]

also would
if currency.find({"id": id}) == True:
work to see if that id exists?

[hexed estuary]

That's too little info to answer - what database are you using, and what python library to communicate with it?

[weak lantern]

Hey

[mild agate]

ah

mongodb

[hexed estuary]

.find is the most generic method - it returns a Cursor, and it's never None. If you are searching by a unique key like an id (so you can never get more than one result) it's more convenient to use .find_one, in which case if currency.find_one({"id":id}): is valid (if there's no match find_one returns None, which is falsy).

[mild agate]

thanks

[mild agate]

oh also, can mongodb databases be defined like this :

db = client.[database name]
users = db.users```

[balmy parrot]

Hi everyone, I have more experience with PostgreSQL, and I was wanting to start exploring other databases, like MongoDB. If anyone could recommend videos or articles that might help, I'd appreciate it.

[peak fox]

Hi folks,
I have many experience with PostgreSQL, MongoDB, mySQL etc
I am looking for a new opportunity for now

[mild agate]

mongodb doesnt want to start

https://paste.pythondiscord.com/JTKA

[mild agate]

just reinstalled ubuntu

[ionic pecan]

there is nothing in this log that suggests it encountered an error. "s": "i", in this insanity of a "logging format", probably means "severity": "info", so we would need to look for anything with E or above

if mongodb has confusing behaviour, it might be a good sign to use a proper database like postgresql instead....

[mild agate]

was something with the linux installation, it stopped bugging out when i reinstalled the os

[idle cedar]

how to check if row exists while having an efficient query with SQLAlchemy/SQLModel

so far the best query was select(True).where(clause).limit(1)
but if you do session.scalar on this statement it returns None instead of False

i guess this is good enough

[sleek yacht]

Get an - on the sp

[unique sigil]

yo

I have built a database of JoJo's Bizarre Adventure stands by scraping the data from wiki but i don't know what to do with it can somebody help?

[craggy arrow]

https://github.com/jimcarreer/dinao
Dunno if anyone would be interested in this toy library.

[coral wasp]

See #1468524576479641744

[last junco]

hello im trying to learn python so i can attempt to make a chess engine. how much recursion do i need to do?

[unique sigil]

Alot

[unique sigil]

A chess engine is really hard to make.

Although if you keep on learning, you will make it pretty soon.

[versed shore]

Start by implementing the rules before you worry about recursion 😅 there are some very tricky rules to implement, like castling through check (forbidden!), pinned pieces, etc

Also, this channel is not the right place. I think #algos-and-data-structs is better

[weak lantern]

Hey

[exotic bobcat]

yurrrr

[cunning plover]

guys, where can I install mysql database I see mysql AI heatWave .....

god, too complicated, postgresql website better downloaded their database system with sql

[harsh cipher]

are there any good local databases similar to sqlite that work well with NetworkX? something lightweight and local that doesn't require a server-client relationship that can basically store networkx diffs. im designing a game and the only ones i found in the python ecosystem are KuzuDB which AFAIK is no longer maintained actively, and an SQLite extension that's still in alpha development

[versed shore]

Is the networkx diff structure pickleable?

If so, you can just pickle it and store it with sqlite, and retrieve it from sqlite and unpickle

[harsh cipher]

good idea i'll try that out ty

[versed shore]

yw!

[harsh cipher]

i couldn't query a pickled json though could i? cuz part [of the reason of having graph diffs in SQLite is for in-game for the AI to be able to query about not only the overall game state but also like, to query how things have changed over the course of teh game

i could always just go for postgres but that's a PITA to use in Steam from my research

unless I host my own cloud postgres which runs into a whole other slew of issues

cuz the game itself is going to be open source and if you wanna set it all up yourself you're welcome to do that, but i was hoping to follow the redhat busines model of "open source for anone to use but you're paying me for conveince of setting everything up for you through steam as a binary connected to all the services"

[harsh cipher]

i think im just gonna suck it up and roll w/ postgres

[narrow rose]

i need help ! how can i connect my datbase usin python

[visual wadi]

which database are you using? you can search for python libraries that works with that particular database first

[lucid orchid]

It depends on the type if it's SQLite or MySQL or what

[bitter iron]

A new package I'm working on, https://github.com/fruch/coodie, if someone is working with Scylladb or Cassandra, it might be interesting.

[tired talon]

Wow cool

https://github.com/cielecon/dbcake

Try my DB too

I stared your project

Please star my project too

@bitter iron

[bitter iron]

@tired talon it doing much more then I thought

[tender flax]

Maybe interesting to this crowd: tomorrow we are hosting a user group on durable queues https://luma.com/5dhntvem?tk=ct8TA2

This is for our Python durable workflow lib: https://github.com/dbos-inc/dbos-transact-py, backend by Postgres

lmk if that's not a good place to post =)

[subtle pike]

want to create a welcome bot for my Instagram group chat please help me out!

[timid brook]

hmm I am having a very confusing issue here with pandas. I am resuming work on a program that basically parses transaction tables from pdfs to store. Now something about the pdf format has changed which results in this breaking but the actual behavior is not making any sense to me at all.

I have tables that I parsed just fine from the pdf still. and I know this because if I just print out the raw dataframe I see the table data (somewhat jumbled of course but that is normal)
Now in order to unify the various tables for merging and cleanup I run the following

            if len(df.columns) != 5:
                df = pd.DataFrame(columns=["Col1", "Col2", "Col3", "Col4", "Col5"])
                print("column count redone")
                print(f"Col fix DF is {df}")
            else:
                df.columns = ["Col1", "Col2", "Col3", "Col4", "Col5"]

But after this runs that 'print the dataframe' command now returns

Empty DataFrame
Columns: [Col1, Col2, Col3, Col4, Col5]
Index: []

And I do not see why or how the dataframe is now completely empty. That command is only supposed to rename the columns not just purge the whole thing. 🙁

And it is even more confusing that if I use an older PDF it works as it used to. But even then the parsing is still actually working for the pdf side its the pandas part that is throwing a fit for some reason.

[winter dragon]

I have a program that cleans datasets and I need some testers? (still in beta tho) who can I ask to help??

Github: https://github.com/Mohammed-Musab/Lazy-Data-Cleaner/

[solid mulch]

Hi. I'm doing an assignment on sqlite3. Im I want to create a database and populate it with records. I commited the command but the database(ebookstore.db) isn't showing up in the folder. I even tried adding the folder's absolute location but still nothing. Maybe you can recommend a more suitable channel for my request. Please help. Thanks in advance, Here's my code:

import sqlite3
user_option = input('''Welcome to the menu! 
                        Choose an option by typing a number: 
                     1) Enter book
                     2) Update book
                     3) Delete book
                     4) Search books
                     0) Exit''' )


connection = sqlite3.connect("ebookstore.db")
cursor = connection.cursor()

cursor.execute('''
CREATE TABLE book(
               id INTEGER PRIMARY KEY
               tltle TEXT
               authorID INTEGER
               qty INTEGER)
''')

connection.commit()

books = [(3001, 'A Tale of Two Cities', 1290, 30), 
         (3002, 'Harry Potter and the Philosophers Stone', 8937, 30), 
         (3003, 'The Lion, the Witch and the Wadrobe', 2356, 25), 
         (3004, 'The Lord of the Rings', 6380, 37), 
         (3005, 'Alices Adventures in Wonderland', 6380, 37)]


cursor.executemany('''
INSERT INTO book (id, title, authorID, qty)
VALUES
    (?, ?, ?, ?)
''', books)
print("books added")
connection.commit()

[versed shore]

Are you getting an error?

From a quick glance, you need commas between your columns in the create table statement

Also, if your create table statement works once, the next time it won't, because the table already exists. You can add "if not exists" after the word "table" to make it work consistently (unless you decide to add or remove columns at some point)

[half cloud]

Im currently working on a project for school, and I have to encrypt some data that are being stored in my database (PostgreSQL), I'm using FastAPI, SQLAlchemy for ORM, and some other stuff for my backend. Does anybody know how I could do that? Because the data I will be storing is health data I am required to encrypt them in my database.

[solid mulch]

Thanks for getting back to me. I didn't get an error. There were some typos but the main issue was that input statement at the beginning. That user input statement blocked the rest of the code from running. Rookie mistake. But thanks for your input 👍

[timid brook]

Well for some reason it seems that in this new case the pd.DataFrame(columns=) method is actually overwriting the dataframe completely. Which I frankly dont know why I didn't encounter this before because I swore it was here for a corner case in the first place. What is happening is due to some other tiny change in the pdfs the detected column count is consistenty 4 instaed of 5. so instead of the df.column method getting run and renaming the columns the other one runs and just overwrites the dataframe. I didnt think that was doing that before but.. huh.

[versed shore]

doesn't pd.DataFrame create a new instance? I see from your snippet that you read df.columns first, so presumably df is at that point already a DataFrame object. Inside the if statement, you then overwrite df by a new DataFrame

[timid brook]

Yea it is and that certainly where the blunder is and I think the biggest confusion was basically why I did it that way in the first place and how I never ran into that issue before. I suppose the point was to infact purge dataframes that did not have the data I needed because in the testing a lot of extra junk is captured but only the 'valid' data had 5 columns. Now the valid data can be 4 OR 5 columns and it was getting hit by the len() != 5 and getting overwritten.

I have created new if catches for 4 columns and 5 columns so for now I will just add a better note as to why I overwite the dataframe in some cases so I can be better aware of the corner cases. I have row data validation done later so I don't currently have issues even some junk comes through but this bug was very odd. And visually inspecting the pdf I cant at all see a difference in the tables so I just need better error handling and tracking I guess.

[versed shore]

I have some experience with parsing pdf in python and hooo boy that's an experience I don't want to repeat. Though I did learn some regex in the process so that's something. So good luck 😄

[timid brook]

Yea the pdf part was a nightmare but I understood it well enough to make it though it requires a lot of customer work for each type of PDF (different financial institutions) but I at least documented that part well enough it is reproducable and adaptable. I thought at first that is what was broken but it turned out to be this column naming shenanigans lol. But now its a perfect learning opportunity to use match instead so Im good with that. All the rest of the program still works how it is supposed to at least 😄

edit for fun this was one of my favorite nonsense hacks needed due to pdf issues

        for row in all_imports.itertuples():
            if row[3][-1] == "-":
                print(f"Minus sign found in charge_name at index {row[0]}")
                all_imports.loc[row[0], "transaction_name"] = row[3][:-1].strip()
                all_imports.loc[row[0], "transaction_amount"] = "- " + row[4]
            else:
                pass

[sharp temple]

Is anyone working as a data engineer here?

[narrow rose]

i use DuckDB or SQL server , postgresql

[narrow rose]

yes i'm data engineer

[cedar tiger]

Are you currently working as one or just "aspire to be" one?

[narrow rose]

both bro

[cold tinsel]

can you help me ..i accidently pushed senstive key into github now i wnt to fix this
how could i do that ?

[storm mauve]

delete or recreate the key in whichever website you generated it
preferably also reset your password in that site

[versed shore]

yeah, it's easier and faster to change the key than it is to scrub it out of git history.

[storm mauve]

even if you try and scrub it out, there is a non negligible chance that it has been automatically copied by someone else, be it for archiving purposes, be it for malicious purposes

once a key has been leaked by any means, you should assume it is now in the hands of hackers and rotate it asap (delete/recreate)

[cold tinsel]

how to do this
could you please help me

[cold tinsel]

is this where i can put my secret keys?

[versed shore]

I think it's under "secrets and variables" in the left hand menu bar

Not sure, I use gitlab

[cold tinsel]

i got it someone told -> secret and variable

[icy glacier]

Let's have a daughter and name her rows.

<@&831776746206265384>

[rich trout]

!cleanban 1415410498161344532 porn bot

[delicate field]

:incoming_envelope: :ok_hand: applied ban to @stable basalt permanently.

[bitter bramble]

Anyone around these parts working with FastAPI, Next JS ? Ive been working on a project and i just dont have anyone working with that stack to talk to lol. Im fairly new to development as a whole and have been working on learning python and a few other languages and am in school for Cybersecurity Minoring in Software Development and focusing on Data Science so i can build Secure AI integrated Infrastructure for SaaS (Security as a Service). please help

[lapis kite]

Should I use sqlite for a data in a project?

[bitter bramble]

it depends on your use case, you can always ask claude to help you research about the topic

[versed shore]

If it's single user, no concurrency, sqlite is a good solution. My suggestion would be to start with sqlite, and if at some point you discover that you need an instanced database, it's fairly easy to replace

[narrow rose]

you use sqlite it's rearly use in big project ,

which approch are best

[delicate field]

Please react with ✅ to upload your file(s) to our paste bin, which is more accessible for some users.

[regal agate]

????

[cold tinsel]

9.0%4
Out[22]: 1.0

-9.0%4
Out[23]: 3.0

how ?

[fierce grail]

Nunca participei dessa comunidade. Sou estudante de ciências de dados.

[fierce grail]

Como posso tirar uma dúvida de um projeto

[grim vault]

Because for an integer division the result is round down (floor()), so floor(-9 / 4) = -3

!e

print(f"{9 // 4 = }")
print(f"{-9 // 4 = }")

[delicate field]

:white_check_mark: Your 3.14 eval job has completed with return code 0.

001 | 9 // 4 = 2
002 | -9 // 4 = -3

[umbral jasper]

Hey, I've got a question about a database.
I was brought in late on a project that uses python and sqlalchemy to set up an SQL Server database that is hosted on Azure.
I need to change the database schema to add a column and an 'on_delete'.
Is it safe to do this or would doing this delete my data? I'm fairly sure it's safe because I'm only adding new things and changing how future interactions go, but you can't be too sure with these things.

[cedar tiger]

Replicate the database on your local machine and then make changes to it as you go. See what happens. I would think sqlalchemy have some sort of migrations scripts to ensure reproducing the same result but I'm not sure.

Oh and of course, have db backups before you deploy

[versed shore]

you can do SQLAlchemy migrations using Alembic. I don't think that SQLAlchemy concerns itself with on_delete triggers, but if you're adding a column and you want to use it in the python application, you'll have to generate a migration or do the necessary code changes yourself

[sly tiger]

hello guys, is anyone familiar with graph based databases here, to be specific neo4j and or semantic (Jena/RDF)

[keen minnow]

what would you ask these people if they were to exist and be here with us?

[sly tiger]

It's for a research project we are working on in our lab - we were trying to model using relational db but found it to be unsuitable, so we need someone who is good with graph based ones to help us experiment and model the data. The data is complex and not a direct syntax question, so didn't ask here since it would an essay instead of a message lol

[keen minnow]

I mean, regardless, you would have to write down that essay at some point.
So you could start writing it so you could point people at it and save time for everyone

[sly tiger]

wait that does make sense, let me flush it out and send over the issue link here then - tq for helping btw

[keen minnow]

I may or may not be online by that time. But that's the risk of waiting for someone to be there.
Though I might check tomorrow morning

[sly tiger]

here we go, https://github.com/isrl-research/sandbox-research/discussions/8 thank you for pushing me to flush it out

I will update with the necessary datasets / data we have in few minutes

[keen minnow]

you were not joking 😅

[lucid warren]

Generate SQLAlchemy query optimization options (selectinload + load_only) from simplified field selection syntax. 😁 https://github.com/allmonday/sqlalchemy-load

[austere gulch]

https://tenor.com/view/napoleon-dynamite-wave-bye-gif-5121301879489036543

um

Hello?
I am koriean
I want make an automatic equity investment program but, I'm new to Python
Can anyone tell me in Korean?

[umbral jasper]

That seems super advanced and way too difficult for a first program.

[weak maple]

Idk korean but ill say this learn the basics of python

Ans then whatever your project is

Idk

[steady gate]

twin i'm korean

형

자동투자 만드는거에요?

아님

어떤거 만드시는 거임요?

저 태크좀요 타임존이 한국이 아니라서

[steady gate]

@austere gulch

[keen minnow]

!rule 4

[delicate field]

Rules

4. Use English to the best of your ability. Be polite if someone speaks English imperfectly.

[steady gate]

Mb comrade

[steady gate]

I will try to use english as much as possible

[keen minnow]

nice!
There are plenty of non native english speakers! So don't worry about your english and do your best!

[steady gate]

thank you sir

[inland sinew]

I have a dataframe, and a pivot table, I'm trying to merge them based off of week_num the issue if I'm correct is that week_num is the index for the pivot table, and a column for the dataframe.

Note: I Only want VWAP and ATR Columns to be merged, not the rest of it

Realized I only need close prices (that's what the pivot point uses)

So here's the new dataframe

Here's what I was originally trying to do:
** - Make a new column called ATR_4, which gets the ATR for Thursday, and only has the values on Thursday**
- Use ATR_4 As my merge, instead of ATR and VWAP as a whole, this way I can get the ATR Data I want (only for thursdays) then do the same for VWAP after

[inland sinew]

Nvm I was able to fix it myself

[long garden]

Hey can I ask why I can't curl in cPanel? I've installed curl when executed in a python script it doesn't work, but if its manual it can like it

curl https://site.com but its only can put it manually, i can't put the in python script anyone can help me?

[versed shore]

curl is not a python builtin. if you want to perform an HTTP request in python, I would suggest using urllib.request.urlopen() from the standard library, or the 3rd party Requests library

also, you're in the channel for questions about databases. if you want to follow up the conversation, continue in #networks

[stray sleet]

what are the scope of dbms

[pseudo plank]

Is there a way to make a leaderboard on an offline code like on VS code?

[nocturne glen]

yes

[long garden]

Oh mb, sorry idk too, i was in a hurry yesterday and I was confused

[lofty moon]

Can you help me write the code for this?

[scarlet shell]

I dont know if im answering this the right way given this is a databases chat, and also I'm very new to python (and programming in general), but my initial thoughts would be to turn the ticket into a string since strings are iterable. Then for letter in string (each digit in the ticket), if digit == "5", fives += 1

Something like that

Sorry for digit in ticket** (represented as letter/character in string)

[somber ember]

hey guys how's the interview process for data engineers these days? is it still code and knowledge heavy like 3 years ago, or is it shifting to "use AI and prove your efficiency and ability to create pipeline" ? also, what'll interview process be like for mid level data engineer

[empty trout]

Is there any way one could connect their python backend to postgres db without the use of an external library? Are there any built in modules that let you connect the two?

[cedar tiger]

unfortunately you would at least need pyscopg2-binary installed to communicate to postgres db from python. If you wanted a database to connect to, without installing an external library, you can look at sqlite instead

!d sqlite3

[delicate field]

sqlite3

Source code: Lib/sqlite3/

SQLite is a C library that provides a lightweight disk-based database that doesn’t require a separate server process and allows accessing the database using a nonstandard variant of the SQL query language. Some applications can use SQLite for internal data storage. It’s also possible to prototype an application using SQLite and then port the code to a larger database such as PostgreSQL or Oracle.

The sqlite3 module was written by Gerhard Häring. It provides an SQL interface compliant with the DB-API 2.0 specification described by PEP 249, and requires the third-party SQLite library.

[empty trout]

thanks, ill check out psycopg2

[empty trout]

ok so i ran into a problem with psycopg2, so I am developing a simple CRUD FastAPI app and I tried to connect it with postgres with the below code:

@app.get("/")
async def root():
    return {"Hello": "World"}

@app.post("/book", response_model=Book)
async def add_book(book: Book):
    cur.execute("""INSERT INTO books VALUES ($1, $2, $3, $4);""",[book.id, book.title, book.author, book.description])
    return book

connection.commit()
cur.close()
connection.close()

The problem is that the connection closes even though the api is still running, I assumed I would run into this problem but wanted to try it out anyways, is there a workaround for this? Is there a way to keep the connection open until the server shuts down?

[blissful wharf]

guys i just discovered an open-source project called GFS The idea is to treat your database like code every run happens on its own branch, and if something goes wrong you just delete the branch while production stays safe. curious if anyone here has tried something like this

https://github.com/Guepard-Corp/gf

[cedar tiger]

This leads to a 404. 🤷‍♂️

[cedar tiger]

git worktree does the same thing

[blissful wharf]

https://github.com/Guepard-Corp/gfs

[blissful wharf]

check it out now

[cedar tiger]

is it any different from git worktree?

[lofty moon]

Thanks

[winter tide]

hello, I have a question about relational databases

let's say I'm desigining the amazon web store
and I'm designing a table for the shopping card
cart
and I need an ID and items

each user can only have 1 cart
so in the items table we have the products
so ID name, description and price
those are our columns
let's say that the carts table is a table where every row is a cart having an ID and items
how do you do the items though
cause if you have multiple items
I don't think you can put a list there in a SQL database
if for example cart with ID 1 has
a tv, a computer and a toilet
how do I put those 3 items in one entry for one row
either we have a table where the same cart appears more than once an each time only one item shows up

[cedar tiger]

You could put each item as 1 row together with 1 cart id

Where:
10 = TV
11 = iphone
12 = charger

Each unique cart is handled by its own specific ID tied to 1 user. So if user ID is 1, then their cart ID is 1. If User ID is 2, then their cart ID is 2.

cart_id | item_id
1       |  10
1       |  11
1       |  12
2       |  10
2       |  12

[winter tide]

that's what I was thinking

would that be bad practice?

I'm not sure if it'd be better to use an extra table called cart_items

[cedar tiger]

I mean if you want, you could look at existing ecommerce site framework and research how they handle or design handling cart data

like woocommerce

[winter tide]

nonono, what I meant is that I agree with your approach but I am inexperienced so I don't know if it's bad to have it designed like that because my brain goes like "well..maybe it's wrong to have multiple rows with the same cart"

[cedar tiger]

I don't think its a bad idea. I mean that's what databases are for. Structure it in such a way that its easy for you to look up, insert, update, and delete data whenever.

so its just a matter of your design, if it makes sense to you at all or not

[mild agate]

what would the code be to get a full SQLite index?

[acoustic folio]

How can I get a Python developer room/group?

[upbeat marsh]

you need a room/group first

[acoustic folio]

did u mean I should create a room/group?

[upbeat marsh]

Yes

[acoustic folio]

Oh, thanks, brother

[untold olive]

Hello all

[lapis iris]

Has happened to take the clf foundations exam

[winter dragon]

hey can you try my program??

I am trying to find bugs - like date flipping bug that still fixing

https://github.com/Mohammed-Musab/Lazy-Data-Cleaner/

latest feature is the log system accessed by `

though it need bit of update, more messages/details.

[hard dragon]

Anyone on here every try building there own database and memory management system from scratch for a program instead of outsourcing or likning a project to an extention of sorts?

[iron fable]

Still need help?

[winter dragon]

i need help

[hard mica]

With what

[hard mica]

SQL DB?

[spice furnace]

Hi everyone, my name is Adnan. I’m currently learning Python, HTML, and CSS on my own. I’m still a beginner, but I’m making steady progress. Glad to be here.

[winter dragon]

program that clean datasets, there are presets and custom one but i need help debugging

[iron fable]

Like to start one? Or one you already have?

@winter dragon

[winter dragon]

I already made one though still in alpha for beginners, here link

https://github.com/Mohammed-Musab/Lazy-Data-Cleaner/

[iron fable]

Is it a script or a program?

[winter dragon]

program with fully working GUI

[hard dragon]

Welcome to the club pal.

[winter dragon]

i still need help

[bleak marten]

Learn one at a time

[vocal crypt]

Y es

Ur right

[frank grail]

I love writing code for my exam on an rdp connection that doesnt have xampp so i cant even test it

Atp im just praying it works 😭

[halcyon mason]

are you by chance türk?

[stark cloud]

Hello there, I have made a rough database schema for a gym management system, I am not very sure on what would be best to do here.

The schema is designed to esnure that one user can enroll in one class, has one exercise plan and has one diet plan. Similarly a single diet plan, exerciseplan and class can be assigned to multiple users.
One trainer can manage multiple exerciseplan, dietplan and class.

Is this schema suitable for this situation? what would you change here?

This is a previous schema I made, isn't this wrong ?

[lime orchid]

hello guys

is this code good

import random
import string

user_info = []

pool = string.ascii_letters + string.digits
random_code = ''.join(random.choice(pool) for i in range(30))

user_name = input("Enter name: ")
user_password = int(input('Enter Password: '))
print('made an account Succefully!')

user_info.append("User_ID: " + random_code +   'Name:' + user_name)

for login_attamps in range (3):
    guess = input('Enter your name: ')
    if guess == user_name:
        guess1 = input("Enter your password: ")
        if guess1 == user_password:
            print('Login was succecfull')
        break
    print(f'You have' {2 - login_attamps}  )
else:
    print('Login got denied. Please try again!')


print(user_info)

[coral wasp]

Each class will have exactly one user?

[stark cloud]

Each user has one class

[gilded sinew]

Where will you implement it

[stark cloud]

Mysql this isn't the final one, looking to implement this one

[gilded sinew]

I am interested in seeing your implementation of code. Because i am actually practicing sqlite.

[stark cloud]

Oh okay I'd be happy to do that, i will share the database when I finalise the schema

[gilded sinew]

Thanks, notify me on completion

[coral wasp]

Each user has exactly one class? THey can't register for multiple classes or take multiple classes over time?

But anyway, your class table has a "user_id", which means it can only have 1 user. You also have a class_id in the user table, so the user can only have 1 table. Modelling wise, this is not really how you want to design a 1-1 relationship, because you can end up with a logical inconsistency (where each direction gets out of sync).

[stark cloud]

Assumption is that one user can only join one class at a time since we are modeling a gym sysyem

[stark cloud]

Oh thanks for pointing it out, the user_id in class table isn't going to be used. I forgot to make that edit in this schema

And the relationship between class and user is that one user can join one class but one class can have multiple users

[coral wasp]

Ok. This seems fragile to me (the idea that a user can't be in multiple classes), but whatever, you'll probably want a history table to track this.

[manic plover]

Hi everyone! any feedback on pip install omna · https://omna.dev/ · https://github.com/gaurjin/Omna
will appreciate if you guys can use and let me know if it helps! thanks!

[stark cloud]

Ohh, would It be better implement a many to many relationship with a seperate table?

[coral wasp]

If that's your goal, sure. It seems reasonable that at a gym a student may take multiple classes, either at same time or at different times of the year.

[hybrid spindle]

Any one know which would be better for an website:- Supabase, Firebase, Cloudflare, AWS. ??

[analog moth]

I personally use AWS for hosting in general, i dont know if id trust their db's but they do offer a very neat free plan raging from 3 months to up to even a year, i think cloudflare has some free permanent plans tho. About quality i think they are mostly the same but i personally think firebase has a more tight ecosystem than any of those other options.

[elder cipher]

MongoDB

[visual ginkgo]

Hello guys, does anyone know base url for api as I have to practice requests library

[true stratus]

Supabase if you want the fastest start. AWS if you need full control. What are you actually building?

[hybrid spindle]

a complete system like instagram + Amazon
Supabase is very slow for api calling

[cedar tiger]

There are a lot of public APIs that you can practice with. https://github.com/public-apis/public-apis

[olive beacon]

based

[nocturne glen]

supabase is slow?

[hybrid spindle]

yes it is also low customizeable

[nocturne glen]

what customization option do you miss

[hybrid spindle]

have you used Firebase

[nocturne glen]

a little bit

[polar swan]

Firebase is the best according to me due to its generous free tier with high speeds

[digital yew]

it's free for general users

later the pricing goes very high with scale

[torn eagle]

Helloo

I made a fun site to try out your sql skills

It's a mystry solve game

can i share the link?

[prime palm]

hey ,guys! what is your opinion about using raw SQL's vs ORM inside an app?
what pros/cons have you encountered while using both approaches?
when one approach is superior over another?

would be glad to hear all your opinion/experience, hope y'all are doing well!

[prime palm]

you may share it in #1468524576479641744 thread

[minor venture]

I am using asyncpg. I need to run multiple select statements within a transaction and compile the data. As they are only select statements I dont need to send and receive each individual statement. In theory I could just send all the statements at once then receive all the responses at once. This in theory should cut down on latency between the database and my code. Is this a worthwhile optimization, does anyone know if asyncpg is setup to handle this I cannot find anything like that.

This is a rough example of what I am doing currently

mylist = []
db.start_transaction()
mylist.extend([A(x) for x in db.fetch("SELECT ...")])
mylist.extend([B(x) for x in db.fetch("SELECT ...")])
mylist.extend([C(x) for x in db.fetch("SELECT ...")])

[thorny anchor]

the database itself may optimize these selects if they don't depend on each other, but there is no mechanism to do parallelism within a transaction. if you want parallelism, you need multiple connections

[minor venture]

Thanks

[burnt vapor]

Need multiple connections i think 😉

[quasi gazelle]

Good day everyone I’m an undergraduate student and I’m looking for a new job, a referral will be much appreciated

[cunning badger]

In case of multiple connections your app can read inconsistent data since they are reading from potentially different db snapshots

[glad plume]

Who has any good email scraping tool please dm

[cedar tiger]

!rule illegal to help as this could lead to email spams everywhere

[delicate field]

Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

[thorny anchor]

it depends on the database and on the isolation level

[cunning badger]

But isolation level applies only to the data within a single transaction. It doesn't guarantee you the consistency of the data between multiple transaction.
So even if you set it to SERIALIZABLE there might be a transaction between two of the reads that changes the data you read.

[thorny anchor]

yes, you're right. it is always possible

[meager marsh]

r u developer? if it's true, can u teach me plz

[nocturne glen]

you open vs code, you type, you hit debug button

[meager marsh]

for what?

[nocturne glen]

to develop code?

[meager marsh]

I alreay know it. How can I learn pytion?

[nocturne glen]

by reading a book, reading the docs, pick libs you wanna use to build something blowing the mind

draft out a flow diagram to solve a specific problem and turn it into python code

[meager marsh]

perfect!. thanks

[nocturne glen]

and when you struggled for a day and you wanna throw the computer out of the window because ai joked on you the whole day, then you always can get help here

[meager marsh]

yeah. AI is really difficult me. thanks ur helping. can i send to u friend request?

[nocturne glen]

if you wanna 👌

[meager marsh]

I sent

[prime palm]

hi guys!
when dealing with postgres timestamps , should i explicitly convert each timestamp timezone from app to UTC on app side or i can simply carry on postgres doing it for me?

[fair cloud]

As long as the timestamps carry timezone information then you could just store them as is. Any queries that filter on that column will be properly handled by postgres. I know some people like to have things homogenous (all same timezone), but I personally don't really care.

Often the representation layer will often convert to user local time anyways and the only reason to have it all in UTC is just because it looks nice.

With that being said, where will the majority of entries even originate from?

[prime palm]

I sometimes can get "Unix epoch" timestamp , user may input some timestamps and the question was (assume i have validated user's input to be a valid datetime obj) so do i need to manually pass like this:

def unix_to_utc(unix_time: Union[str, int]) -> datetime: # kind of bad example Unix timestamp is already in UTC

    return datetime.fromtimestamp(int(unix_time), tz=timezone.utc)

def str_to_datetime(raw_dttm: str) -> datetime:
    return datetime.fromisoformat(raw_dttm)

or i can be sure postgres will do it correctly for me?

[prime palm]

as i got you the recommended way is to store every timestamp with tz in UTC (it would be done automatically by postgres) and to display timestamp in user's timezone? Btw, i must put server's timezone to UTC?

[fair cloud]

If I understand correctly, are you asking if you can just write the unix epoch directly or convert it to a datetime object before?

[prime palm]

no, i mean should i put tz explicitly to a datetime in app before inserting this timestamp to db or i can do nothing and delegate it to postgres

[fair cloud]

I guess that depends on the ORM you are using (if you are using one). I guess you could just try both ways and see if there 's a difference.

However, regarding server timezone: As you mentioned, Postgres will always store the timestamp interally as UTC. "If you dont provide a timezone in the input string, then it's assumed to be in the time zone indicated by the system's TimeZoneparemeter. " (docs)

So the server's timezone only matters if you dont provide an explicit timezone, but this can cause problems with double offsets, so just ensure that timezone is always present and you should be good

[prime palm]

got it, thanks for note about server's timezone.
when i will be dealing with timestamps output to user I must set session's timezone appropriate to user's? I am using SQLalchemy ORM for my web app , does it support this manipulations or I can do it another way?

[opal dawn]

i'm trying to optimize back-end code to faster serialize some nested response data and considered using orjson. but fastapi docs seem to discourage it as unnecessary, and wondering what, if there are any exceptions for using it: https://fastapi.tiangolo.com/advanced/custom-response/?h=orjson#custom-response-class

[grim jackal]

Hey everyone. I’ve been practicing strict data cleaning and just finished a project matching exact crypto news publication times to 1-minute market data (Kaggle link: https://www.kaggle.com/datasets/yevheniipylypchuk/bitcoin-news-vs-1m-btc-price-action-2025-26).

The hardest part was standardizing the UTC timestamps and handling the exact T0/T+15m delta calculation. If anyone here has experience building backtesting pipelines or scraping financial news, I’d love a quick roast of the methodology in my notebook. Did I miss any obvious edge cases?

[torn sphinx]

Db topic isnt active ? :(

[torn sphinx]

just newly created

o

np

[light summit]

Redis is pretty lit ¯_(ツ)_/¯

[torn sphinx]

mongoDB

shelve

sql

[prime torrent]

NOO

RethinkDB

is the best

[deft badge]

Agreed @prime torrent

[open gull]

Ok

[rancid rampart]

Yes?

[open gull]

Im sorta being more a teacher than a helper

welp im not too fussed

[rancid rampart]

Well, it's nothing bad ^^

[open gull]

ill try to not tell you everything, like ill teach you some SQL and how to use python to get stuff from databases and you can integrate that into your thing in your own way

[rancid rampart]

Okay ^^ I'll have some issues here and there, but i guess it'll work out, somehow.

[open gull]

oke so get your console up

[rancid rampart]

Give me a second, just finishing to upload everything

[open gull]

oke

[rancid rampart]

And i guess installing discord.py on the site too

Oukay, that didn't work out so well

Ready @open gull, so i'll go in the MySQL cmd, rite?

[open gull]

yup

[rancid rampart]

I'm in

[open gull]

oke

the syntax to create a table is:

CREATE TABLE(name datatype, name datatype, etc.)

datatype for string isnt string tho

its text

so

CREATE TABLE(name text, password test)

[rancid rampart]

So CREATE TABLE(statsDb string, inventoryDb string) ?

Oh

[open gull]

and people mostly put the CREATE TABLE and other commands written capitalised

they dont matter

[rancid rampart]

okay

[open gull]

you can put create table(name datatype)

[rancid rampart]

But is it string or text?

[open gull]

or cReAtE tAbLe etc

text

[rancid rampart]

Okay

Done

[open gull]

also

just realised

i was wrong

[rancid rampart]

So if i understand it correctly i now created 2 empty tables?

Oh..

[open gull]

whoops

.<

[rancid rampart]

Well, now you can teach me how to delete them!

[open gull]

:P

[rancid rampart]

😄

[open gull]

im kinda confused why that worked tho

[rancid rampart]

REMOVE TABLE(name datatype) ?

[open gull]

no

[rancid rampart]

You never now :p

[open gull]

i mean you did the thingy wrong tho

cos i told you the wrong thing

[rancid rampart]

Yeah

weird enough

I mean, even if i type random sht in it, it doesn't complain

[open gull]

CREATE TABLE tableName(columnname datatype, columnname datatype)

what did it say when you put that first command in

[rancid rampart]

Nothing lol

http://puu.sh/xvLfr/fd24728d6b.png

[open gull]

OHHH

I FORGOT TO SAY

[rancid rampart]

.-.

[open gull]

to finish a line of it

you put ;

[rancid rampart]

Oh

I gotta get some errors here lol

Yup 😄

[open gull]

yeah :P

[rancid rampart]

So CREATE TABLE statsDb(copper number, silver number)

I figured since it was text instead of string, it might be number..

[open gull]

so you can do something like
CREATE TABLE playerStats(playerID int, copper int, silver int, gold int, etc)

[rancid rampart]

Darn now i gotta look up all the things i need

So, level, exp, max level, max exp, mana, max mana, etc. too ?

[open gull]

yup

you can add columns later tho

[rancid rampart]

CREATE TABLE playerStats(playerID int, level int, copper int, silver int, gold int, exp int, maxexp int, mana int, maxmana int, stamina int, maxstamina int, hunger int, maxhunger int, fatigue int, maxfatigue int)

I think i got everything 😄

Done

[open gull]

yup oke

make to type ;

[rancid rampart]

Yeah i've done that

Query OK, 0 rows affected (0.03 sec)

[open gull]

good good

now i will tell you how to add a row, or entry, or item

call it whatever i guess

[rancid rampart]

Okay

By the way, thank you for teaching and helping me so much, even tough you probably have better things to do.

[open gull]

INSERT INTO tableName VALUES (value1, value2, value3, value4) you have to put a value for every column

[rancid rampart]

Urgh

[open gull]

so thats 15 values i think

[rancid rampart]

Well, what do i put in for playerID?

[open gull]

just i guess 0?

it doesnt really matter at the moment really

[rancid rampart]

Oh, okay, that gonna be a lot of 0 in that table then

[open gull]

you can have another table which correlates things like playerID and discordID

and ofc more information i guess if you need to

[rancid rampart]

INSERT INTO playerStats VALUES (0, 1, 0, 0, 0, 0, 125, 100, 100, 100, 100, 100, 100, 0, 100);

Done

[open gull]

at you could do another 1 or two with playerID 1 and 2 just for testing purposes

[rancid rampart]

done

[open gull]

and now i will tell you how to get values, which you will use in python later on

[rancid rampart]

Okay?

[open gull]

SELECT columnname, columnname, etc FROM tableName WHERE condition

the column name can be one or multiple

or it can be *

which means all of them

and WHERE and beyond is optional

without it it will just give you everything

[rancid rampart]

So SELECT playerID FROM playerStats WHERE playerID = 1?

Oh

Okay

Gotcha

[open gull]

well that would only return a playerID

[rancid rampart]

Yeah

[open gull]

most of the time people would use *

[rancid rampart]

And how would i get an entire row then?

[open gull]

use *

SELECT * ...

[rancid rampart]

SELECT * FROM playerStats WHERE playerID = 1 ?

Nice

[open gull]

yup

i think the spaces between playerID and = etc

[rancid rampart]

No it works

[open gull]

wont work

oh oke

good good

[rancid rampart]

And without the WHERE .. it just tells me everything

Which is cool

[open gull]

yup

[rancid rampart]

I guess in python it'll be returned in an "array" ? So like [0,0] = 1 since the first row and colums are the playerID 1

Or not?

Well, that isn't even relevant i think

[open gull]

well i think

its like some weird object

[rancid rampart]

Haha ^^

[open gull]

but you can do

sqlthing = sql.getsomethingidk
for i in sqlthing:
    print(i)

[rancid rampart]

Oh

Niet

[open gull]

and i is like

a list?

idk ill find it out before telling you

[rancid rampart]

Well i is a number that increases from 0 to null i think

null beeing the end of the list

[open gull]

ok so you will want a python module called MySQLdb

but it cant be downloaded using PyPi, aka pip

wait hangon hm

i just realised you need root access to install this and you dont have root access

[rancid rampart]

Well, if there is a mysql database, there is most likely something like it already installed, is it not?

[open gull]

ah yeah

it says its already installed

[rancid rampart]

See? 😄

[open gull]

so you should be able to do import MySQLdb in your code and it should work

[rancid rampart]

I'll just delete the data structure on my inventory file, and reupload it

[open gull]

import MySQLdb

db = MySQLdb.connect(host="localhost",    # your host, usually localhost
                     user="john",         # your username
                     passwd="megajonhy",  # your password
                     db="jonhydb")        # name of the data base

# you must create a Cursor object. It will let
#  you execute all the queries you need
cur = db.cursor()

# Use all the SQL you like
cur.execute("SELECT * FROM YOUR_TABLE_NAME")

# print all the first cell of all the rows
for row in cur.fetchall():
    print row[0]

db.close()

here is the example i found

host and user are found on your database page

password is what you set

and db is default i believe

[rancid rampart]

i created a new one

but username i the accounts username, right?

[open gull]

or i think name$default

well if thats what is shown on the database page then yeah

it is my username for me

[rancid rampart]

Okay done

And uploaded

I just don't remember how to start a python script in the console .-.

[open gull]

cd /path/to/dir/where/python/file/is

python3 main.py

[rancid rampart]

Traceback (most recent call last):
  File "main.py", line 11, in <module>
    from commands import cmd_ping, cmd_inventory, cmd_dsix, cmd_dhundred, cmd_stats
  File "/home/Blade67/commands/cmd_inventory.py", line 12, in <module>
    db="RpgDb")        # name of the data base
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")```

[open gull]

hm

[rancid rampart]

Maaaaybe because i had the bot running locally too z.z

Mh, nope

[open gull]

hangon a sec

i think

you have to put username and $ and then name

as the name

so like

[rancid rampart]

I might have tiped the password wrong too

[open gull]

username$database

[rancid rampart]

Nope

[open gull]

send me your code without the password

[rancid rampart]

``import discord
import random
from discord import Embed
import MySQLdb

em = discord.Embed(title='Inventory', description="", colour=discord.Color.gold())

db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="Blade67", # your username
passwd="password123", # your password
db="Blade67$RpgDb") # name of the data base

cur = db.cursor()
cur.execute("SELECT * playerStats")

for row in cur.fetchall():
print(row[0])

db.close()

async def ex(args, message, client, invoke, sender):
await client.send_message(message.author, embed=em)
``

[open gull]

change the host to the one on the database page

[rancid rampart]

python3 main.py
Traceback (most recent call last):
  File "main.py", line 11, in <module>
    from commands import cmd_ping, cmd_inventory, cmd_dsix, cmd_dhundred, cmd_stats
  File "/home/Blade67/commands/cmd_inventory.py", line 12, in <module>
    db="RpgDb")        # name of the data base
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1044, "Access denied for user 'Blade67'@'%' to database 'RpgDb'")```

[open gull]

try changing the database name back to username$database

[rancid rampart]

python3 main.py
Traceback (most recent call last):
  File "main.py", line 11, in <module>
    from commands import cmd_ping, cmd_inventory, cmd_dsix, cmd_dhundred, cmd_stats
  File "/home/Blade67/commands/cmd_inventory.py", line 15, in <module>
    cur.execute("SELECT * playerStats")
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 217, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 378, in _query
    rowcount = self._do_query(q)
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 341, in _do_query
    db.query(q)
  File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use n
ear 'playerStats' at line 1")```

[open gull]

ah you are connected, but just you formatted the execute statement wrong

[rancid rampart]

oh

[open gull]

SELECT * FROM playerStats

not SELECT * playerStats

[rancid rampart]

For me defense, you sent me that 😄

It works now ^^

End it sent a exception lol

[open gull]

oh sorry :P

i just ripped it from stack overflow

[rancid rampart]

Traceback (most recent call last):
  File "/home/Blade67/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 601, in _create_direct_connection
    local_addr=self._local_addr)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 695, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.5/asyncio/base_events.py", line 682, in create_connection
    yield from self.sock_connect(sock, address)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 439, in _sock_connect_cb
    raise OSError(err, 'Connect call failed %s' % (address,))
ConnectionRefusedError: [Errno 111] Connect call failed ('104.16.59.5', 443)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/Blade67/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 304, in connect
    yield from self._create_connection(req)
  File "/home/Blade67/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 578, in _create_connection
    transport, proto = yield from self._create_direct_connection(req)
  File "/home/Blade67/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 624, in _create_direct_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno 111] Can not connect to discordapp.com:443 [Connect call failed ('104.16.59.5', 443)]```

[open gull]

woah that is a big traceback

[rancid rampart]

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "main.py", line 41, in <module>
    client.run(DATA.TOKEN)
  File "/home/Blade67/.local/lib/python3.5/site-packages/discord/client.py", line 519, in run
    self.loop.run_until_complete(self.start(*args, **kwargs))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/home/Blade67/.local/lib/python3.5/site-packages/discord/client.py", line 490, in start
    yield from self.login(*args, **kwargs)

  File "/home/Blade67/.local/lib/python3.5/site-packages/discord/client.py", line 416, in login
    yield from getattr(self, '_login_' + str(n))(*args, **kwargs)
  File "/home/Blade67/.local/lib/python3.5/site-packages/discord/client.py", line 346, in _login_1
    data = yield from self.http.static_login(token, bot=is_bot)
  File "/home/Blade67/.local/lib/python3.5/site-packages/discord/http.py", line 258, in static_login
    data = yield from self.request(Route('GET', '/users/@me'))
  File "/home/Blade67/.local/lib/python3.5/site-packages/discord/http.py", line 137, in request
    r = yield from self.session.request(method, url, **kwargs)
  File "/home/Blade67/.local/lib/python3.5/site-packages/aiohttp/client.py", line 555, in __iter__
    resp = yield from self._coro
  File "/home/Blade67/.local/lib/python3.5/site-packages/aiohttp/client.py", line 198, in _request
    conn = yield from self._connector.connect(req)
  File "/home/Blade67/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 314, in connect
    .format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno 111] Cannot connect to host discordapp.com:443 ssl:True [Can not connect to discordapp.com:443 [Connect call failed ('104.16.59.5', 443)]
]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe80eefaf98>```

Thats the whole thing

[open gull]

ah ok

it cant connect to discord

[rancid rampart]

Duh

[open gull]

does the site allow that?

[rancid rampart]

Idk

[open gull]

like it allows hosting websites

but does it allow to connect to others?

[rancid rampart]

https://www.pythonanywhere.com/forums/topic/7384/

Only via http access...

[open gull]

oh

so you need to pay to use more than whats on that whitelist?

[rancid rampart]

Yeah...

That sucks

[open gull]

welp

how much is it?

$5? i know cheaper

[glacial fractal]

SESTREN Help Manual

Command not found.

Syntax: $help [cmds...]

__Shows help documentation.__

$help: Shows the help manual.
$help command: Show help for a command
$help Category: Show commands and description for a category

Type $help <command> for more info on a command. You can also type $help <category> for more info on a category.

[rancid rampart]

5$ a month

[open gull]

oh whoops

[rancid rampart]

But i'm broke, i wouldn't use a free provider if i wasn't

[open gull]

i know a site that 3 euros a month

for a virtual server

[rancid rampart]

Like i said, i'm broke

Like

Not a cent broke

[open gull]

which means full access to a computer that runs all the time, so more than just python

root access :)

[rancid rampart]

Infact i'm in negative

[open gull]

rip

i guess thats annoying

[rancid rampart]

Yup

Well

I guess i can get a vps....

Let me check.

Its good to have people indepted to you.

[open gull]

indebted*

:P

[rancid rampart]

Oh, my bad

English isn't my first language, so bare with me ^^

[open gull]

oh really?

i genuinely didnt realise

[rancid rampart]

I'm French ^^

[open gull]

youre pretty good at english you had me fooled :P

[rancid rampart]

Well, i did learn it for roughly 4 years

But believe me, France's english level is extremely low

[open gull]

ive been learning german for over 4 years and im not that fluent :P

[rancid rampart]

Like "Hello, how are you" is godlike here

Oh, i am

Fluent in German i mean

small talk while waiting for my vps space to be set up for me

[open gull]

:P

[rancid rampart]

Gotta love the good old minecraft times, where you hosted servers for others on a 600€/month dedicated server 😄

I was such a nerd.

Welp, still am

[open gull]

:P

[rancid rampart]

I just told him that i'd need some space on his server for the bot with full access, he just says "k" ...

Oh please tell me his server runs debian D:

Well, he mostly does it for me because he wants me to join his administration team z.z

[open gull]

:P

i keep replying with :P

whoops

[torn sphinx]

😛 😛 😛

Im happy

so happy

so so so

happy

Wrong chan oops

[open gull]

id offer you hosting on my server but i sometimes stop running it

[rancid rampart]

Its fine

Hes setting it up

[rancid rampart]

Okay

[rancid rampart]

Sorry that it takes so long D;

Ouh no, he gone D:

[viral crag]

I feel like someone should do a database overview post

Maybe I'll get around to it in a bit

[polar willow]

what database would you guys reccomend for a discord bot (that potentially will have a lot of writes)

[light summit]

I've been using Redis and it's done pretty well for me. In memory NoSQL. A lot of people have recommended Mongo, but I've yet to check it out.

[polar willow]

i find it weird that most of them are servers. why would i need that when only the bot will be retrieving from it?

[viral crag]

@polar willow that's kind of how almost all databases work

You cannot do asynchronous filesystem reads and writes

The GIL prevents that

Most filesystems prevent that

You need a server

I would recommend MongoDB using the Motor library

[viral crag]

I'd recommend it because there are no asyncio ORMs yet and Motor has a very easy and nice API

[polar willow]

thx i'll give it a go

[split raven]

When would sychronous sqllite with sharding be a bad idea ?

I'm a noob with databases and haven't figured out much yet with asynchronous calls

Currently using the sychrnous SQLite for my bot( hosted on one server)

[viral crag]

@split raven it would always be a bad idea

Actually that's an incredibly craptastic idea

You shouldn't have more than one process ever accessing an sqlite database at once

[polar willow]

But how can we get more craptastic?

[split raven]

Ohh ok

If you don't mind me asking, how would you use shaeding then with SQLite?

I know nadeko has sharding with their database

[viral crag]

You wouldn't

Don't use SQLite

lol

You need to be using some database server

eg, mongodb, postgresql, etc

[deft badge]

rethinkdb

[viral crag]

Haven't seen a single project that uses that

[split raven]

@viral crag https://github.com/Kwoth/NadekoBot

the database is sqlite 🤔

and there's sharding in the code 🤔

but maybe the sharding is for their global bot? and the local sqlite database is for self hosters

[viral crag]

Ok look

there is a simple way to settle this

http://howfuckedismydatabase.com/

[split raven]

that doesn't answer how nadeko is doing it though - they are an established bot with like thousands of users

i'm just trying to understand how it works for their bot

from what you said, my guess is that the sharding is for their global bot that anyone can invite to their server

[viral crag]

they're probably using an sqlite daemon

[split raven]

and the sqlite database is for self hosters

i seee - how does that resolve the issue of sqlite and concurrency though?

[viral crag]

Well the daemon would load the entire database into memory

and then everything else would ask the daemon to retrieve and write data

so you never have more than one process writing to the file

it's still a shit way of doing it though, don't get me wrong

I tend to recommend this to people not too familiar with SQL: http://motor.readthedocs.io/en/stable/tutorial-asyncio.html

but if you're OK with writing SQL, there's https://github.com/aio-libs/aiopg

or, god forbid, https://github.com/aio-libs/aiomysql

there is no asyncio for sqlite

the entire format is just useless for concurrency

[split raven]

oh ok, will check out the links

thanks

[torn sphinx]

@deft badge hi

so i have a few questions about rethinkdb

[deft badge]

@torn sphinx ih

What questions

[torn sphinx]

hihi yes first could i have a code snippet to see how you handle the connections

cause that side note screenshot is getting me worried ill have to fuck around with multithreading

[deft badge]

I don't and it works fine

[torn sphinx]

ah kk

bye

code snippet needed btw

[deft badge]

import rethinkdb as r
conn = r.connect()

r.db("dbname").table("hihi").insert({"Dank":"memes"}).run(conn)

[hollow kite]

hello guys here i can ask about problems in the sql installation error?

[hollow kite]

i have problem to restore my db in microsoft sql

https://i.imgur.com/Zk2WtNB.png

[viral crag]

The error message explains what the problem is and how to solve it

[hollow kite]

dont work i try different microsoft sql and dosnt work

[viral crag]

then you're using the wrong one

this is the wrong place to ask tbh, this channel is about Python database drivers

[umbral sapphire]

Sql is a database right?

[viral crag]

SQL is Structured Query Language

[umbral sapphire]

Uum, lel

Haha

[near cradle]

the topic should be updated if you can't actually ask about databases in this channel.

[torn sphinx]

why?

[near cradle]

because the topic states "you may ask questions or talk about databases in this channel"

[torn sphinx]

and the content of channel?

[near cradle]

sorry?

I was refering to what @viral crag about this being a channel about Python database drivers. that's fine but the topic says otherwise, so that's pretty confusing for users then.

[viral crag]

I dunno, I thought that's what it was about

[near cradle]

https://giphy.com/gifs/yosub-girl-taco-why-not-both-3o85xIO33l7RlmLR4I

[gusty fossil]

What SQL library for python would you guys recommend? It's my first time doing this so I have no clue what to use

@ me if you can

Thanks

[viral crag]

@gusty fossil SQLAlchemy

But it doesn't support asyncio

If you need that then currently your best option is MongoDB with Motor

[gusty fossil]

Ah okey

Appreciate it, thanks

[gusty fossil]

Anyone here got a good resource to learn SQLite from?

[viral crag]

SQLite is not something you use in production

that said

you usually use it via python's standard dbapi

and the SQL is very similar to MySQL

[gusty fossil]

Yeah I figured, I just found the documentation for it so its all good. Im using this since I cba to install a whole new library

I just need to store some data for a discord bot, so i think it should be fine?

[viral crag]

absolutely not

use a database server of some kind

sqlite will lock up your bot, it has no concurrency

[gusty fossil]

Oh well shit

[hot crest]

Do we have someone familiar with Lets Encrypt & MongoDB combination? I need help as it keeps screwing me over 😄 Help would be really appreciated

[viral crag]

I have experience with both, but not together

I'm assuming you're getting some SSL error?

[hot crest]

yup 😄 SSL: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown Been trying different ways and all of them seems to lead into same spot no matter what :S

[viral crag]

Looks like an unknown cert

What're you doing here, using certs for authentication?

[hot crest]

well trying to secure connection to database and using server certs on authentication

[viral crag]

You could secure the actual connection with an SSH tunnel if you wanted, but I digress

Have you worked with Mongo auth before? It's finnicky at best

[hot crest]

well what comes to basic --auth parameter setupping, permissions and so on I have worked with them

[viral crag]

It honestly just seems like your certs aren't stored correctly on the server

Just from the error

[hot crest]

also what makes it interesting

openssl verify -CAfile shows mongodb.pem being OK 😄

[viral crag]

The file itself might be fine, but..

Actually, how does Mongo store certs again? Surely you should have an actual cert store

[hot crest]

yes I'm giving it parameters for CA & PEM file

[viral crag]

Those aren't cert stores though

Those are actual certs

[hot crest]

mongo actually wants the files

[viral crag]

Are you specifying the host option?

And have you checked that your config matches your parameters or at least isn't overriding?

[hot crest]

yup

[viral crag]

Are you just using this to secure the connection? I forget whether mongodb can also auth with a cert

Since if you don't need to validate a client cert then you don't need to specify a CA

You just provide a valid cert for encryption and the client will use it when the ssl option is specified - like https

@hot crest

[hot crest]

sry currently on phone so lacking behind

[viral crag]

I actually need to go shortly, but consider whether you need to be validating client certs, and if so, ensure the client certs were signed with the same CA you're using for the server

If not, don't use a CA file

[hot crest]

Okay, thank you

[torn sphinx]

I have user.execute("UPDATE USER set bought = " + bought + " where ID = " + str(id_) + ' ') sqlite3.OperationalError: near "1": syntax error when I use user.execute("UPDATE USER set bought = " + bought + " where ID = " + str(id_) + ' ')

any suggestions on how to solve it?

<@&267630620367257601>

[deft badge]

yes

[viral crag]

jesus christ

you hate security don't you

lol

[deft badge]

enclose the 1 in '

also

[torn sphinx]

why?

[viral crag]

do you know what SQL injection is?

[deft badge]

please please please escaoe your sqilite

[torn sphinx]

there is no 1

[viral crag]

id_ is 1

(clearly)

[torn sphinx]

or is it bought?

but anyway

[viral crag]

but yeah, joph is right

never just insert stuff into the string

[torn sphinx]

it is a private database

[viral crag]

doesn't matter

[torn sphinx]

does that make a difference?

oh

[viral crag]

here, I'll fix it for you

one second

user.execute("UPDATE USER set bought = ? where ID = ?", (bought, _id))

wait, didn't see the other one

there

[torn sphinx]

ok thanks

i'll try and see

[viral crag]

Do you understand what that does?

[torn sphinx]

it'll probably work

yeah it uses the ? and inserts the values

[viral crag]

in order, yep

this is the safe way to do it

[torn sphinx]

ok thank you

and it was id_ but thanks for the code

[viral crag]

well, you get the idea

[torn sphinx]

yea

[blissful tapir]

Use pickle

[viral crag]

get out

:P

[blissful tapir]

e.e

[silent coral]

does someone know a good guide for basic in sqlite3 and python and discord.py combined or so?

atleast some good sqlite3 stuff

like the data types, syntax and so on

because i want to combine my bot with sqlite3 and damn im a noob in this

[high gust]

sqlite3 doesn't work with discord.py -> It's not asynchous

[timber stratus]

ahah

[silent coral]

wut

which sql language does work then?

with discord.py

[high gust]

@silent coral https://github.com/timofurrer/awesome-asyncio#database-drivers

[viral crag]

that's not entirely true

you can use it with d.py

but it'll slow your bot to a crawl most likely

I believe there are async drivers for it but they don't come with python

[silent coral]

tbh: does it really need to be something with asyncio? I imported asyncio since the beginning of developing my bot and it is still unused

(marked in grey)

[viral crag]

ideally, yes

[silent coral]

damn i just wanted to save data so my bot doesnt have to scrape the website every time and now im getting confused with asyncio and stuff

[timber stratus]

What is asynio again?

asyncio*

[silent coral]

18.5. asyncio — Asynchronous I/O, event loop, coroutines and tasks

still dont know

[viral crag]

think of it like poor mans' threads

it gives you the illusion of multiple things happening at the same time

[timber stratus]

poor mans?

[silent coral]

oh

aaah

[timber stratus]

wat

illusion?

[viral crag]

yeah

it's not true concurrency

need me to explain in more detail?

[timber stratus]

yes pls

[viral crag]

ok, well

as most of these frameworks are, asyncio is backed by an event loop

[timber stratus]

k

[viral crag]

you can define async functions - with async def - these functions return Coroutines

[timber stratus]

yup

[viral crag]

when you await a coroutine, you're essentially putting a task onto the event loop

[timber stratus]

ahah

[viral crag]

and then the event loop has time to go and do something else

and then come back to your task

it's pretty fast when done right so you usually don't notice

[timber stratus]

yeaaahhhhhhhh, just bare in mind that im 13...

sooooooo, wat???

[viral crag]

so essentially, all you're doing is pushing tasks onto the event loop

they're executed one at a time

[timber stratus]

i made a discord bot once but i dont really understand async

[viral crag]

it's just very fast

[timber stratus]

wats the event loop?

[viral crag]

pretty much exactly what it sounds like

it's the main loop of your program

[timber stratus]

oh

[viral crag]

it's constantly running and executing the tasks you pushed onto it

but it can only do one task at a time

[timber stratus]

ooooooooooooookkkkkkaaaayyyyyy

[viral crag]

when you push a task onto it, you're returning control to the event loop so it can go and do something else

[timber stratus]

wait so if i do ```python
async def func():
print("Hello")

[viral crag]

No

[timber stratus]

wat

[viral crag]

Well it wouldn't be much use if it repeatedly executed each task would it