Yu Haolun

Order Trade System Development

Spearheaded the development of new features for the global Order Trade System, enhancing system functionality and user experience

ISO27001 Auditing

Facilitated ISO27001 auditing processes in Europe by demonstrating programming workflows and ensuring compliance with international standards

ELT/ETL Pipeline Migration to Azure Databricks and Snowflake

Successfully migrated the ELT/ETL pipeline from a private cloud (Trino) to Microsoft Azure Databricks and Snowflake, enhancing the ingestion of complex AVRO data structures and reducing costs. Key achievements include:

• Leveraged Databricks and Snowflake's advanced capabilities to streamline the ingestion of intricate AVRO data structures, ensuring efficient data processing and analysis.
• Developed an AVRO message send/receive function integrated with the AVRO Registry Center, enabling compressed data handling and significantly reducing bandwidth usage.

This migration optimized the ELT/ETL pipeline by leveraging cloud-based technologies to handle complex data structures, reduce costs, and improve overall system performance and scalability.

Optimized legacy OLAP platform performance

Significantly improved maintainability and reduced the burden on the Trino cluster

This optimization improved the system's stability running in the private cloud and accelerated query speed for end-users by reducing the burden on the big data platform.

Infrastructure Development - Gateway, Registry Center, Monitoring

Implemented a robust infrastructure design for 'North-South' and 'East-West' traffic management and request tracking, handling over 1 million requests daily.

• Developed Lua scripts for OpenResty and maintained Consul Cluster/OpenResty in DEV/PROD environments, customizing the gateway with regional-based blacklists and specific redirection for internal users.
• Developed a service to auto-register/deregister Docker services to/from the Consul Cluster by monitoring docker.sock, enabling microservices interaction and registration as downstream services on the gateway.
• Deployed and maintained Cloud Host monitoring with Prometheus and Grafana, enabling automatic alerts for Cloud VM crashes.