Datetime with timezone support

[fluid star]

Hey,

I'm working on schedules for my project but want to make sure to abide by users timezones. How exactly can I do this?

So in settings I have TIME_ZONE = "GMT"; USE_TZ = True but that's kind of different to what I want.

I have a HTML an input like this:

<input type="datetime-local" name="date_time" />

But it's giving just a datetime back and I'm not sure how to get that specific users timezone. I either want to convert ALL times to UTC just to keep them consistent for schedules, or extract the users timezone out, which would have to be like Europe/London.

How can I do this?

Thanks, I can provide more detail if I missed anything

Datetime with timezone support

[frosty granite]

when you say it's giving just a datetime back, what exactly do you mean? How is that different from what you expected?

[fluid star]

Well it's a datetime-LOCAL input so I thought it might idk give me a timezone with it, but that wasn't the case. I'm not sure how the LOCAL part fits into it, and how to extract that out. I could have a dropdown input next to it and let the user select a timezone, but that means i need to provide a list of the 100s of timezones there are and it just seems a pain (and no other site really does this)

[frosty granite]

what exactly is it submitting in the POST?

why do you say "that wasn't the case"?

[fluid star]

Good point... let me print it out, one sec

'date_time': ['2024-03-08T15:41']

[frosty granite]

hmm

that is indeed bad

[fluid star]

So just dateTtime, no like timezone

yeah

[frosty granite]

how did you get that output?

[fluid star]

I've never had to actually use datetimes before, only stored them so i've never even had to think about this

[fluid star]

Just printed request.POST

Oh yeah good point, im not using django forms

[frosty granite]

dunno if that matters

try changing datetime-local to datetime

[fluid star]

okay

[frosty granite]

hm, it turns the form into a plain text form

[fluid star]

oh yeah, there isnt actually a datetime input (without -local)

[frosty granite]

I wouldn't use that at all

[fluid star]

(Also mini note, im only using the datetime in the backend really, so its not something i can just convert on the frontend)

I'm just looking into django docs for timezones atm seeing if anything is there

[fluid star]

oh really?

[frosty granite]

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/datetime-local strongly suggests it doesn't "do" time zones

also I don't recall ever seeing one of those input fields, ever, in decades of using the web

so I'd look into django forms

[fluid star]

Ah i had a fear of this

[fluid star]

Yikes

[frosty granite]

https://docs.djangoproject.com/en/5.0/ref/forms/fields/#datetimefield

I doubt they're hard to use

as you might guess, I've never used them. But Django seems to do everything sensibly.

[fluid star]

Doesn't mention about timezones on there though

Seems like it gives the same result '2006-10-25T14:30Z'

[frosty granite]

that's good

the Z is the time zone

[fluid star]

Oh right

[frosty granite]

my rule is: never use naive datetimes

[fluid star]

is a naive datetime one thats a set timezone and never changes timezone?

[frosty granite]

no

it's one that has no timezone at all, and is thus ambiguous

[fluid star]

I mean all im doing with this is: user chooses a date and time, django calls api with that datetime. But then the backend defaults to UTC and then the users input isnt followed along unless they are UTC too

[fluid star]

Ah okay

[frosty granite]

what do you mean by "the users input isn't followed along"?

[fluid star]

users timezone*

[frosty granite]

how can you tell?

[fluid star]

Actually thats a good point, the default is UTC and my time is UTC so i'm just guessing. but there's no evidence towards it being the same because django from what i can see never gets told what the users timezone is in the input

[frosty granite]

still don't know what you're talking about

you seem to fear that django is discarding information

I doubt that it is.

You need to describe in detail what you're observing -- not your guesses -- to explain why you think it's discarding information

I do expect it to discard the user's browser's time zone when it saves their time into the database. But that's OK since it'll get added back later if you display it in the browser.

and it's also OK because the time that it enters into the database will be unambiguous -- at least if you've got USE_TZ=True, which you should.

[fluid star]

Yeah, i mean i dont care about the frontend atm, that's the worst part cause i know the frontend is good with that, but I just want to make sure the api call as soon as the user submits either passes the Timezone with the request or changes the time to something like UTC

[frosty granite]

🤷

I have no idea what API you're referring to

[fluid star]

Well ill change my PC timezone and see if i can send these schedules again and see if it just magically "works", hopefully it does

[frosty granite]

if you do things the Django way, and have USE_TZ=True, it will not throw away information.

That should be all you need.

why are you changing your PCs time zone? Just as a test?

[fluid star]

Yeah, cause my time is UTC (i believe at least lmao) i cant really tell whats happening, so if i change to something with a bigger time difference i can see if the API is using that

[frosty granite]

ah ok

I was afraid you thought you needed to permanently change the time zone in order to host your web site

there's got to be an easier way to test, though

[fluid star]

Also by the way, how can i compare the time? this is where my initial error came from which i forgot about

date_time_to_obj = timezone.datetime.strptime(date_time, "%Y-%m-%dT%H:%M")
date_time_to_obj.timezone.make_aware(date_time_to_obj, timezone.get_current_timezone())
if date_time_to_obj < timezone.now():
    # send right away

I tried this but it's not working very well

[fluid star]

Yeah true, im not sure

[frosty granite]

what does "not working very well" mean?

if it's complaining about comparing naive to non-naive, that suggests that you don't have USE_TZ=True set

[fluid star]

Just all the code is a dump, the datetime to obj doesnt even have .timezone so i get AttributeError: 'datetime.datetime' object has no attribute 'timezone'

I dont know how any of it is even meant to work cause i dont understand what i even need to do

Cause im guessing timezone.now() is using my USE_TZ from settings (so GMT) which is the server default one, so I need to convert date_time_to_obj to that? But then we're back at the problem of im not sure what timezone it is to begin with

Nvm the bug was fixed by changing date_time_to_obj.timezone.make_aware to date_time_to_obj = timezone.make_aware

[frosty granite]

I don't really understand what's going on, but I can't think of a good reason you'd ever need to call make_aware

if you're getting timestamps from django, and you have USE_TZ=True, they should already be aware

[fluid star]

cause im comparing timezone.now() which is aware to something that isnt

[frosty granite]

why are you dealing with a naive datetime in the first place?

fix that

that's a bug

see above under "never use naive datetimes"

[fluid star]

Yeah, i dont know what the alternative is

[frosty granite]

the alternative is to use aware datetimes everywhere.

[fluid star]

how can i get time now without a tmezone and without knowing the users timezone

[fluid star]

isnt that what im doing though

[frosty granite]

why would you even want to do that?

[fluid star]

now im confused lol

[frosty granite]

"get time now without a timezone" violates my rule 🙂

I think your problem is: you don't know how to get an aware datetime from the user

[fluid star]

Well the idea is that timezone.now() is aware, and ive got

date_time = data.get("date_time") which isnt aware, so im converting it to TRY and be

date_time_to_obj = timezone.datetime.strptime(date_time, "%Y-%m-%dT%H:%M")
date_time_to_obj = timezone.make_aware(date_time_to_obj, timezone.get_current_timezone())

I think your problem is: you don't know how to get an aware datetime from the user

Yup, exactly lol

[frosty granite]

ok why is data.get("date_time") not aware?

that's a bug

fix that

don't try to convert the naive time into an aware time. Instead, avoid ever getting the naive time in the first place.

Capisce?

[fluid star]

HTML datetime-local isnt giving me the aware, thats why, like i say im not sure how to pass in that timezone with the input

[frosty granite]

true.

that's why I suggest you stop using datetime-local.

Use django forms instead.

[fluid star]

But surely using django forms just for 1 date-time input is excessive, there's got to be an easier solution?

[frosty granite]

sure -- continue struggling with timezones

[sick dagger]

To get the user's local timezone for this kind of thing, there are libraries that help with that: https://github.com/adamcharnock/django-tz-detect

[frosty granite]

afaik those are the two options

[fluid star]

Cool, yeah something like that seems perfect. Arguably even more excessive, but seems like an easy solution and out of the way once added for future use

[frosty granite]

Ben's idea should work. I personally dislike it because it will require Javascript -- and since I've never used django forms, I might be underestimating their difficulty. Still, they're what I'd try first.

[sick dagger]

It's not excessive, because there really isn't a way to tell what timezone a user is in until you run something in their browser like JS

[frosty granite]

does django-forms require JS?

[fluid star]

yeah but not on my end though i dont think, just loading it in at the start of base
e.g.

{% load tz_detect %}
{% tz_detect %}

[sick dagger]

I genuinely don't know what you mean by "django-forms"

[frosty granite]

nor do I 🙂

[sick dagger]

How is Django's forms supposed to do that?

[sick dagger]

Oh

Then why are you suggesting it? 😆

[frosty granite]

I was hoping you'd know 🤣

seriously https://docs.djangoproject.com/en/5.0/topics/i18n/timezones/ sure looks like it will do The Right Thing when collecting a timestamp from the browser

since django is pretty slick, I assume it really will; and if that's the case, I'm wondering how

[sick dagger]

When you enable time zone support, Django interprets datetimes entered in forms in the current time zone and returns aware datetime objects in cleaned_data.

[frosty granite]

sure

[sick dagger]

The default time zone is the time zone defined by the TIME_ZONE setting.

The current time zone is the time zone that’s used for rendering.

You should set the current time zone to the end user’s actual time zone with activate(). Otherwise, the default time zone is used.

So you have to call activate() with a specific time zone for it to know

[frosty granite]

so how does it know "the current time zone"? Presumably that's the browser's time zone, not the server's

hm

[sick dagger]

Yes. But on the server you have to call activate with it

And in order to do that, the browser has to tell the server somehow what the time zone is

... and that's what django-tz-detect does

[frosty granite]

huh

[fluid star]

Ah, so on first request it seems that on first request it stores the users timezone (with js) to the DB session?

[sick dagger]

It literally provides:

• a view that accepts a timezone and sets it on the session
• JS that sends the timezone to that view
• middleware that calls activate with that timezone

[frosty granite]

this is turning out to be more complex than I'd expected! Good thing I don't do front-end stuff 🤣

[sick dagger]

Correct (or whatever session backend you're using, not necessarily DB)

[fluid star]

Ah yeah, cool

To be honest i knew one day i'd have to deal with timezones and it wouldn't go well, the same with trying to implement currencies! They were just as bad, but i ended up going the custom route for that 😕

[sick dagger]

I've linked to the relevant sources above from the library, it's really quite "simple"

Timezones, locales, currencies, all awful to deal with 😅

[fluid star]

Yeah lol, maybe I should've stuck to local customers, im trying to branch too much

[frosty granite]

I wonder if Stripe has some helpful library for all this stuff

[fluid star]

Would that be if i was using the stripe as the api im calling? Cause i'm using AWS for this

[frosty granite]

well Stripe is just about payments afaik

[fluid star]

yea

[frosty granite]

AWS is a zillion different cloud services

[fluid star]

ohh like transactions

So like the date of them

[fluid star]

Yeah my bad, i thought you thought i was using the stripe as the api im calling from the backend

[sick dagger]

I imagine Stripe likely puts dates in UTC in all its API info

[frosty granite]

any professionally-designed API would

[sick dagger]

Exactly

[frosty granite]

they tend to use JSON with iso8601 timestamps in UTC

[sick dagger]

Only ever convert from a user's timezone and to a user's timezone at the very boundaries of your interaction with the user

[frosty granite]

amen brudda

[fluid star]

So does that mean the frontend converts their input to UTC before posting it

[sick dagger]

I would imagine so, yes

[frosty granite]

i.e., maintain the smallest possible amount of code that deals with naive datetimes

[fluid star]

ah

[fluid star]

Yeah that would be ideal

[sick dagger]

Same as data encoding, convert to/from bytes only on the very edge

[fluid star]

im using HTMX though so I get little flexibility

[sick dagger]

That's not true 🙂

an HTTP request coming in is the edge

[fluid star]

Im not sure how i'd convert the inputted time to UTC with htmx though lol

[frosty granite]

wouldn't be surprised if HTMX had a way

[fluid star]

Actually good point, there might be discussions on htmx about that, ill take a look in a minute

Worst case i could use something like Apline or the forbidden Hyperscript or even just JS to somehow convert to UTC without showing that the user

[frosty granite]

at this point, Ben's suggestion of django-tz-detect seems best

[sick dagger]

The django-tz-detect method is only one way, you could also store that information with JS in a hidden field in the form and send it along that way, but you then wouldn't be able to call activate before form processing, so the date from the cleaned form data will be in the server's time zone and you'd have to do some maths on it

The biggest downside to that library is "the first page your user hits won't know their timezone"

[fluid star]

Ah yeah that sounds like pure pain... Yeah i think you're both right to go with the django tz library method

[sick dagger]

But by the time they come to POST a form, it'll have been set on the session by the JS

[fluid star]

yeah i saw that a moment ago, shouldnt be an issue though

[fluid star]

perfect!

[sick dagger]

Good luck 🙂

[fluid star]

Thanks man, both of you in fact, appreciate the help!

[sick dagger]

Time zones are horrible, but I enjoy wrapping my head around them 🙃

[fluid star]

Yeah that's it, sometimes i dont like to just go with a solution i dont understand, it's fun to dive into how and WHY it works, but by next month when i need this agian ill be back questioning how I done it 😆

[frosty granite]

try hard to do it right the first time; it'll save time down the road

[sick dagger]

Wise words

[frosty granite]

the trouble is: sometimes quick-and-dirty is the best 😐

and telling them apart takes ... decades

[fluid star]

Would you recommend me to load tz_detect in my BASE file, or just in this one page where I need it?

{% load tz_detect %}
{% tz_detect %}

I don't want to be making extra calls or slowing pages if it's not needed

[frosty granite]

as early and as often as possible

so that it's available to as many of your pages as possible

SO THAT YOU DON'T HAVE TO THINK

[fluid star]

lol that's not a bad idea!

[sick dagger]

Yeah, it's not a huge overhead

[fluid star]

To be fair datetimes are actually used in a few places but just not handled very well, so i might want to re-think them at the same time

[fluid star]

Cool, i mean as long as it's not slowing anything down then should be fine

[sick dagger]

It wouldn't be, it's out of band

[fluid star]

Oh yeah it's asynchronous!

[sick dagger]

All that tag does is include a tiny bit of JS: https://github.com/adamcharnock/django-tz-detect/blob/develop/tz_detect/templates/tz_detect/detector.html

And even then, it only includes it if it hasn't already set it on the request, so... it will only be called once per session.

[fluid star]

Oh yeah, that's cool then. i'm not sure how well that'll work with my deployment though, as we accept AWS S3 as a storage

and the library wont be in the storage, and wont be served locally...

hmm

[sick dagger]

What do you mean?

As long as collectstatic and {% static %} etc works, it doesn't matter where the static files are hosted

[fluid star]

Oh cool, collectstatic will update even with library dependencies

thats fine then

Tbh collectstatic doesnt work for me on prod, so that could end up being a pain but ill just have to debug and fix it lol

ive been putting it off for a while

[sick dagger]

collectstatic will work on any Django apps (self-written or library pip-installed) that are in INSTALLED_APPS

[fluid star]

It just crashes when run collectstatic from prod lol, and i couldn't be bothered to copy all of the environment variables manually to local

im not sure the best time to collectstatic, on deployment pipline or on startup of container?

[sick dagger]

Well, you'll have to figure that out at some point before you go live 🙂

I always run:

• manage.py migrate
• manage.py collectstatic
  on container startup, just before starting gunicorn

[fluid star]

Eek... yeah we've already got production out in the wild, but it's fine at the moment the only static files we need are in there as Collectstatic used to work.

[fluid star]

Yeah that's what I have at the moment, cool (well without collect static)

[frosty granite]

fwiw I have been trying, and failing, to get django-tz-detect working 😦

[fluid star]

Hm so i've added the library, and set my timezone to UTC -9 and it's still what i can see not picked it up:

2024-03-08T07:45
{'success': False, 'message': 'Date time cannot be in the past'}

    date_time = data.get("date_time")
    print(date_time)
     date_time_to_obj = timezone.datetime.strptime(date_time, "%Y-%m-%dT%H:%M")
    date_time_to_obj = timezone.make_aware(date_time_to_obj, timezone.get_current_timezone())
    if date_time_to_obj < timezone.now():

But i think the issue is that i'm trying to make it aware, but the whole point of this is that it's meant to already be aware.

[fluid star]

Me too!

I've followed the docs to step 6, its posting the new timezone changes to my session fine, but the backend just isnt getting that "timezone" support

[frosty granite]

huh, mine is failing tests, and then (if I remove the tests) crashing when I render the page

https://gitlab.com/offby1/django-tutorial/-/commit/f0cfdc3e1075e6f70c92b159451c8b08bda2341c

[fluid star]

Oh damn

[frosty granite]

I don't know enough about URL handling to debug it

[fluid star]

yeah i've not got any errors, its just not doing anything really

What's the actual error?

[frosty granite]

django.urls.exceptions.NoReverseMatch: Reverse for 'tz_detect__set' not found. 'tz_detect__set' is not a valid view function or pattern name.

[fluid star]

Ah, i just used path('tz_detect/', include('tz_detect.urls')),

dont include a name="tz_detect"

Though I think i've missunderstood what this library is meant to do. I thought it was meant to convert the datetime just before the post, but i dont think it does that?

It says that "As a result, dates shown to users will be in their local timezones." which I need the opposite

[frosty granite]

doesn't make any difference, I tried both ways

[fluid star]

It does a good job at that ^^, but doesn't work for POST requests... hmmm

[fluid star]

oh yikes

But unless I'm wrong, this isn't the right library to solve the issue we're trying to solve

[frosty granite]

aha! got it

[fluid star]

Nice!

[frosty granite]

I bet you could use this for POSTs, too ...

[fluid star]

You'd hope so right

[frosty granite]

the library puts the user's time zone in their session. When you get the POST data, you could glue the time zone to the datetime

somehow

[fluid star]

Yeah that's true, i wonder if they have a way to automatically do this

[sick dagger]

It really should alread

From their docs

[fluid star]

Ben did mention they use activate() which is meant to be that

yeah

[sick dagger]

When support for time zones is enabled, Django stores datetime information in UTC in the database, uses time-zone-aware datetime objects internally, and translates them to the end user's time zone in templates and forms.

and forms

Are you using Django's forms?

[fluid star]

Ah.. does that mean django forms... nope

[sick dagger]

Well… 😅

It's not going to magically know to convert any string in POST data that happens to look like a datetime

It's only when you specifically use a form with a DateTimeField and have activated a specific time zone

[fluid star]

oh yikes

I cant find that code in their repo though

[sick dagger]

It's Django that does that

All the library does is call activate for you

If you want to do it without Django forms, then you'll need to call Django's timezone utility to get the current time zone, then use that when converting the string in request.POST into a datetime object

[fluid star]

So this wasn't far off

date_time_to_obj = timezone.make_aware(date_time_to_obj, timezone.get_current_timezone())

[sick dagger]

No, it wasn't!

[fluid star]

timezone is from django, so i'm guessing get_current_timezone uses tz_detects session one?

[sick dagger]

But Django's forms do it for you. You could look at the DateTimeField source to see how it does it

[fluid star]

Ah i see

[sick dagger]

It uses whatever one is "activate"d

[fluid star]

Cool so django-tz library should override that

Ah yeah django uses this:

 def to_python(self, value):
        """
        Validate that the input can be converted to a datetime. Return a
        Python datetime.datetime object.
        """
        if value in self.empty_values:
            return None
        if isinstance(value, datetime.datetime):
            return from_current_timezone(value)
        if isinstance(value, datetime.date):
            result = datetime.datetime(value.year, value.month, value.day)
            return from_current_timezone(result)
        try:
            result = parse_datetime(value.strip())
        except ValueError:
            raise ValidationError(self.error_messages['invalid'], code='invalid')
        if not result:
            result = super().to_python(value)
        return from_current_timezone(result)

well i shouldn't need to look into djangos code, let me start by debugging mine and ill hop back here if i cant get it to work

Ohh i know what i'm doing! I'm calling make aware on the USERS timezone (which it is already that cause of django tz), i need to call make aware for DJANGOs timezone, cause im converting the time to a standard SET time

Woo, it works!!

date_time_to_obj: timezone.datetime = timezone.datetime.strptime(date_time, "%Y-%m-%dT%H:%M")
print(date_time_to_obj)
date_time_to_obj = date_time_to_obj.astimezone(pytz.timezone("UTC"))
print(date_time_to_obj)
if date_time_to_obj < timezone.now():
    return error # error

# console
2024-03-08 08:12:00 # in users TZ (in this case UTC-9)
2024-03-08 17:12:00+00:00 # converted to UTC