I get that some specialisation is useful, but I would like to see more T-shaped data folks.
I think the specialisms are really helpful for hiring - data engineering and data science involve a lot of different things so it's a lot easier to hire someone if you don't require skill in both.
That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.
The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.
Me too. There were no data engineers when I was a baby data scientist (15 years ago now).
I think that this was a terrible idea (the splitting out of roles). I remember once being asked in an interview if I was able to pull my own SQL or would I need an analyst. That kind of split always struck me as absurd, given that the only way to build a decent model is to know the data very well first..
> That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.
I honestly feel like this is driven by the titles. I will note that most of the unsuccessful/bad data scientists I've met were the ones who only cared about the models. As I always said to people, I can run sklearn in a loop over model families if that's all you bring.
> The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.
This is probably just a good signal in tech in general, some curiosity is really required to be successful in this industry.
30 min livebug fix and review for me on a job I specced out? Ticket to another team to review and schedule for next sprint and thus 2 weeks of metadata likely lost or malformed.
These setups look great (but full of invisible friction) when things are going well but shit the bed as soon as you come across the simplest of human errors that needs a solution.
Then the BI people (who had never written a line of code except maybe some SQL) got jealous, started calling themselves "data engineers", broke everything ("what do you mean 'we need tests???'"), and ruined the title.
Back then his bank started building a data science/engineering dept, and he jumped ship, and essentially did the same work for 2x the salary. Let me tell you, understanding Black-Scholes and what the data actually means and how to source it, as well as what finance people actually are looking for is a skill that's 1000x as difficult to get as learning R/Python/Pandas etc.