I don't think writing tests is helpful when you have good practices for code review as well as some sort of canary system in place.
#Hot take: We have *zero* tests at work
spice nacelle
junior canyon
Canary tests are tests
spice nacelle
I guess I should rephrase to developer-written tests then
(We do run production traffic against a staging server and compare responses and system state)
junior canyon
Oh so you test against the user
I think you should have this, additionally with tests. Tests failing in this stage (split off prod traffic) should be treated as critically as something failing in full roll out. Of course the impact is reduced, but this is how we see it where I work
spice nacelle
This might be a shitty engineering practice again but we basically fast-forward 2 weeks of traffic as well as some live traffic onto the staging server and see if things are completely fucked up
junior canyon
Recorded prod traffic, interesting
spice nacelle
What industry do you work in? We're in payments
spice nacelle
I'm sure you guys have put more thought into engineering culture đź—ż
junior canyon
Recorded prod traffic is a good testing methodology actually
If you can ensure it’ll test everything
spice nacelle
We can't
junior canyon
I’m sure you’ll miss many cases but still better then 0 tests
spice nacelle
But if it finds some sort of regression in prod traffic it saves the requests +- 6 hours (+ the ones that are relevant for state) as a seperate thing and then runs all of the known regressions as well
junior canyon
How do you handle auth?
With recorded prod traffic just curious for my own knowledge
spice nacelle
How do you handle auth?
I'd imagine this would be much harder to pull off even for just a small part of Amazon
junior canyon
Yeah I work on identity/auth stuff and there’s sooo many systems for this
And it would be super painful
spice nacelle
I think the biggest factor here is that we have a single source of truth for that
If you want to use our API you create the OAuth client in our system period
So authentication isn't that complicated in our case, the harder part was comparing responses for jobs because you can't always guarantee the exact same runtime on some job running on some other machine, for example report generation etc.
If a customer creates the report, then waits 5 seconds and requests it every 200ms the generation might finish right before request X in production and right after X in staging