#Do I want to be a Data Engineer or something else?

11 messages · Page 1 of 1 (latest)

smoky ice
#

I'm currently employed and I've started putting job alerts on websites for a job that is closer to my interests.

I don't mind learning and using other languages as well but I want the job to focus on Python. For personal projects I've gathered data from through APIs and web scrapping and for work I've created scripts to run tests and gather results. This is the kind of work I would like to do as part of my job.

I don't really have the statistics knowledge or the in depth ML knowledge to do data analysis or data science, plus from my understanding they focus on the collected data.
Data engineering looks like it includes aspects of what I like and I've started going through a coursera course on the subject but so far I'm afraid it would lead to more DBA or devops type work rather than collecting the data.

Is there anyone here whos job is to collect data from varied sources that can give me advice? Are you data engineers or what kind of jobs should I be looking for?

lavish sinew
#

I suppose you can describe data engineering as collecting data from different sources, but it's generally focused on building long-lasting pipelines. So more "let's find a way to get Cassandra data into kafka so we can ingest it into the data warehouse", less "munge data from a web api to do something"

#

There are some just plain ol' development jobs that do that: one team at my company is in charge of gathering data from a variety of external sources and putting it into a common format, and so they're building things like you describe. Or you do that as part of the job in devops and qa-eng roles

#

We don't have an open position at the moment so I can't give you an example job description, but you'll probably unfortunately have to just do a lot of keyword searching and have discussions with recruiters about what you want to do

smoky ice
#

So more "let's find a way to get Cassandra data into kafka so we can ingest it into the data warehouse", less "munge data from a web api to do something"
Yeah, that was the feeling I was getting. I understand storing the data I collect is important but I want to be the one getting the data from APIs, web scraping, files etc and storing it there in the first place.
Those jobs are exactly the kind of thing I'm looking for, I guess if I focus on businesses that collect data with roles for either developers or data engineers I should find something.

proven oasis
#

Data Engineering is Cloud Engineering + SQL

#

If you enjoy SQL, relational DBs, and designing database schemas Data Engineering is for you.

#

IMO the advice I would give you is to not aim for that job

#

Data engineer does not offer a career path into MLE these days. I find that there's a huge divide between "CS jobs" and DE which is very much DevOps