#🔒 Having trouble reading past kinesis records using pyspark, been stuck on this all day

5 messages · Page 1 of 1 (latest)

iron dome
#

So i’m using readStream and specifying to start at a timestamp 15 minutes in the past. i then want to count the records from 15 minutes ago until the current time. My output log shows that it starts reading from that timestamp correctly, but my output count is only like 20-30, when it should be around 180. My whole job (using AWS glue) is supposed to start up, count the records in the past 15 minutes, write it to my s3, and end the job. this has been super hard to figure out

neat valveBOT
#

@iron dome

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
iron dome
#

please

neat valveBOT
#

@iron dome

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.