25 Unpopular and Contradictory Opinions on Data and Engineering
If your Data Product does not generate revenue, it’s an internal tool
Many companies will get more value out of a half-decent Business Intelligence team than they will trying to build Data Products
Data Engineering is a form of Software Engineering
Data engineering projects often do not require the complexity seen in software engineering projects
Complex data engineering projects are not more complex than software engineering projects; they just used a specialized set of tools
Centralized data platforms are good
Centralized data teams often are not
Your data team structure should compliment the rest of the organization
More data analysts probably won’t make your data team more effective but more analysts might help your organization use data more effectively
More data engineers might make your data platform better
“Data” as a standalone function has created more confusion than any other function. UX and User Research might be a close second
Data PMs are only warranted when your product is VERY technically complex
Data PMs are often a byproduct of misaligned business goals
The data infrastructure design and tooling at Meta, Netflix and AirBnb are not going to solve your startup’s problems
Most of your tooling decisions will not matter
A handful of your tooling choices will cause acute frustration
Don’t use multiple vendor tools when one would suffice; the marginal additional value for a dedicated solution is seldom worth the additional complexity
Even the most pragmatic engineers get caught up in the vendor hype cycle
Many companies using Snowflake, BigQuery and Redshift could just be using rollup tables in Postgres
Save a few exception, there is now little differentiation in the Data Warehouses; most novel feature sets are now table stakes
That data catalog you’ve been scoping out is probably a waste of time
A metadata catalog is very much worth it
You don’t need a new tool to have a metadata catalog
Your organization’s Data Strategy has less to do with its data than you think
If you think your Data Strategy is complete, you should revisit it
Ones that resonate most (read: I learned the hard way): 1, 11, 14, 16
I haven't had the same experience with 7 - care to share more?