Big Data on Chain
A Second Brain
In the vast landscape of knowledge and experiences, there exists an undeniable truth — the vast unknown. It's in the gaps of what we don't know that our memories may fade, and valuable insights slip through the cracks. The volume of information can be overwhelming, and the fear of losing what we've learned or read a lot. It's this realization that fuels my motivation to build a second brain and it's a commitment to continuous learning and exploring.
Mapping of Contents (MoC)
| Category | Topics |
|---|---|
| #Data Engineering | Data Engineering, Data Warehouse, Data Architect Spark Optimization, Built DataOps Tool Foundation of SQL , Automation |
| Reading and Writing | Writing Book, Writing Blog |
| #Productivity | Data Platform tool, Workflow |
| Knowledge Package Management | SecondBrain Notes, Thoughts |
| Practices Hands On | Bootcamp, HandsOn, Podcast & Talks |
What I am taking notes...
What we learned Legacy
Legacy technologies such as Informatica, SSIS, and SQL Server are often seen as outdated, but they are still widely used in the big large company (exclude IT company)
How people are interacting with AI ?
Here are some samples from the 100, with one quote for each. The full list is at the bottom of this article
Memory Management During Data Processing
Imports from managing the memory and resource during processing data and programming, Why we need to clean up TempDb in Data Warehouse and how to optimize data pipeline performance?
Learning Vector Database
The motivation of this thoughts that trying to discover the LLMs and Generative AI
Real Learning About SQL
They say that As Data Worker, we need to learn the SQL, They provide the SQL mandatory functionality of SQL scripting, But what do you think about this? Is that good enough?
Improve Data Engineering Hand Book?
This is project and I do need to get feedback, summary of the book, This is markdown project, it is possible to be scanned and loaded into LLMs
Portable Data Platform
Having question why they've been spent to much money for building data platforms (including pipelines, database, frontend, etc ...)
Optimization for report SLAs. There are view created on top of tables which contains heave business logic
Optimize data models inside the view, Using SQL execution plan (Synapse / MSSQL), Data Movement in Cluster
Create a Second Brain with PARA Method
You can use any Tool/Application such as Notion, Note Ever, Obsidian, Code, Apple Notes, Google Note, etc to create your Note Taking System.
Self-Taught
There are things I noted from the inspired video that how we should teach/coach others to do what they wanted to do.
reduce effort we've been spending on Warehouse ?
Spending to large efforts to improve, maintain, build data warehouse. Modern data stack is not resolve the problem.
Semantic layer
Improve data quality: data quality dimension, data reconciliation. Control data quality during digital transformation which is moving data from legacy system to new system
Data Alert Design
Improve data quality: data quality dimension, data reconciliation. Control data quality during digital transformation which is moving data from legacy system to new system
Data Governance Approaches
Think process to resolve data governance problems. How to help company to improve their data, change the culture of working with data.
New existing DLT service
Think process to resolve data governance problems. How to help company to improve their data, change the culture of working with data.
Data Vocab
Question to architect data systems. Data dictionaries