Cliff Crossland, CEO and co-founder of Scanner, dives into innovative uses of AWS Lambda functions for performing fast full-text searches over large volumes of logs and data lakes. Cliff discusses their experiments with different programming languages—such as Python, Java, Go, and Rust—to optimize performance and manage costs effectively. With detailed comparisons of JSON parsing libraries, cold start times, and CPU architectures, this talk showcases how Scanner successfully leverages Lambdas to handle petabytes of log data efficiently. Featuring live Q&A, performance charts, and real-world use cases, this is a must-watch for anyone interested in cloud computing and big data analytics.
00:00 Intro
00:34 Innovative Use of Lambda Functions
03:01 Cliff's Background and Scanner's Mission
03:59 The Scale of Log Data and Search Challenges
05:23 Lambda Functions and Indexing Strategy
08:18 Language Performance Experiments
09:11 Key Takeaways from Language Experiments
12:25 Detailed Performance Results
28:23 Optimizing JSON Parsing in Go
29:22 Exploring Rust for Data Parsing
31:51 Choosing the Right Data Format
36:00 Lambda Functions and Network Performance
38:40 Comparing CPU Architectures for Lambda
40:55 High-Level Takeaways and Recommendations
44:01 Q&A and Final Thoughts
Follow Cliff on Twitter [ Ссылка ]
Learn more about Scanner [ Ссылка ]
Website [ Ссылка ]
Join the Community [ Ссылка ]
Follow us on Twitter [ Ссылка ]
Follow us on LinkedIn [ Ссылка ]
Subscribe on YouTube [ Ссылка ]
