#Data Warehouse

3 messages · Page 1 of 1 (latest)

muted drum
#

Hi all, I have a database design question. How would you go about creating a data warehouse with Django? Let's say I have a Django project called 'shop'. As usual, customers come and buy things. I then need to be able to run stats and complex queries on the data, but I don't want it impacting my 'shop' production database. I feel like I need to clone the 'shop' database or at least keep the databases constantly in sync so I can run my queries reliably. Should I create a read-only replica? What is the best way to go about this? Thanks.

regal galleon
#

Yep, a read-only replica is a good idea. You could also limit how long a SQL query can run before it's automatically killed. I think in postgres it's SET statement_timeout 30s; CrunchyData writes a lot of good content on this. https://www.crunchydata.com/blog/control-runaway-postgres-queries-with-statement-timeout

Crunchy Data
Control Runaway Postgres Queries With Statement Timeout

A well tuned production Postgres database is capable of easily running thousands or up to hundreds of thousands of queries per second on a beefy instance.

#

Streaming Replication is probably the phrase you're looking for regarding the read replica.