How Fast Can You Parse a File with 1 Billion Rows of Weather Data Using Java?
Last January a challenge was posted online by Gunnar Morling: How fast can you parse a file with one billion rows of weather data using Java? Little did I know this deceivingly simple question would lead me down a path that taught me all about: parallelism, memory mapped files, SWAR techniques (SIMD as a register), bit twiddling, branchless code, mechanical sympathy, Graal native compilation and finally... I even turned to the dark side: using sun.misc.Unsafe. Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in >4 minutes, to processing everything in under two seconds. Who knew Java could be _this_ fast?
Transcript
Last January a challenge was posted online by Gunnar Morling: How fast can you parse a file with one billion rows of weather data using Java?
Little did I know this deceivingly simple question would lead me down a path that taught me all about: parallelism, memory mapped files, SWAR techniques (SIMD as a register), bit twiddling, branchless code, mechanical sympathy, Graal native compilation and finally... I even turned to the dark side: using sun.misc.Unsafe.
Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in >4 minutes, to processing everything in under two seconds.
Who knew Java could be this fast?
About the speakers
Experienced developer and architect, robotics enthusiast and hobby mathematician