Qu Haotian logo

Qu Haotian

Data Engineer | Powered by Curiosity

Qu Haotian headshot

Profile Summary

Key Skills

Data Engineering
Python, SQL, Spark, Iceberg, Hive, Delta Lake, Polars, batch and streaming ingestion, Microsoft Fabric, IBM CP4D.
Data Analytics
Power BI, Grafana, Cognos Analytics, Medallion Architecture, Kimball Modeling, requirement gathering.
Infra and DevOps
AWS, data platform design, Terraform, Linux RHEL, InfluxDB, DevSecOps, containerisation, ETL CI/CD.
Soft Skills
Leadership, documentation, Agile methodology, training and teamwork, conducting user workshops.
Languages
English native, Chinese native, Japanese elementary.

Experience

JCO Analytics, Data Engineer

Jan 2023 - Present

  • Built scalable on-prem and cloud data platforms and pipelines across 9 projects in telecommunications, public sector, and pharmaceutical domains using Spark PySpark, DataStage, SSIS, and Microsoft Fabric.
  • Applied Medallion and Kimball architectures across data platform and analytics projects.
  • Led data engineering efforts across 2 projects, collaborating with analysts and DevOps engineers.
  • Developed reusable OOP Python libraries adopted across Fabric-based projects, improving development efficiency and standardisation.
  • Optimised Spark pipelines using Hive, Iceberg partitioning, and shuffling strategy, cutting data query time by 40%.
  • Fine-tuned InfluxDB performance by reducing cardinality by 50%.
  • Implemented monitoring and alerting systems, maintaining zero production incidents over 1 year.
  • Designed cloud data solutions with governance and privacy controls to support compliance and reliability.
  • Created 2 Gen-AI presales demos with IBM chat agents and vector databases to demonstrate RAG capabilities and facilitate client adoption.

Hong Lam Marine, Web Application Developer Intern

Feb 2021 - Sep 2021

  • Developed and maintained an in-house web application front end with Angular.
  • Collaborated with 4 stakeholders to improve usability.
  • Provided IT support, facilitated IT training, and participated in 2 major technology upgrade initiatives across the company.

Client Projects

Hyphen Pharma - MS Fabric End-to-end Data Platform Solution, Data Engineer

Mar 2025

  • Developed config-driven Microsoft Fabric pipelines with Medallion architecture and robust ETL DevOps deployments.
  • Reduced code complexity and maintenance effort by about 20%.
  • Architected the data model for the platform solution.
  • Engineered a parallel, API-driven ingestion framework in Spark with robust pagination handling, backfill capability, and schema evolution.
  • Enabled scalable and reliable data loading for the data platform.
  • Collaborated closely with data analysts to empower self-service analytics.
  • Accelerated Fabric data solution development by 40% through modular, reusable ETL components.

Major Local Telecom Company - Observability Dashboard, Data Engineer

Aug 2024

  • Created PySpark data pipelines processing millions of records per data source daily.
  • Maintained data platform InfluxDB.
  • Built 4 Grafana dashboards with product owners.
  • Interacted closely with stakeholders.
  • Provided co-development sessions to teach 4 client engineers on the involved technology stack.

Education

National University of Singapore

Expected Jun 2028

Bachelor of Technology (Honours) in Software Engineering, part-time. GPA 4.7.

Relevant coursework: C++, x86 Assembly, Data Structures, Linear Algebra, Discrete Structures, Networking, Databases, React and NodeJS.

Singapore Polytechnic

Apr 2019 - Apr 2022

Diploma in Information Technology, specialised in Data Science and Data Analytics.

Relevant coursework: Python, DevOps, Machine Learning, Data Visualisation, Data Engineering, Full-Stack Development.

Certifications