How do you emulate real world hyper-scale load?

And relax; the World Cup is over - congratulations to France on their win – and a huge well done to the England team too. However, the World Cup isn’t just about setting records on the pitch, with the changing ways we consume content the BBC and ITV have been setting records off the pitch for the number of online coverage.

Live match streaming requests at the BBC increased from a total of 15.9m for Brazil 2014 to 56.3m this year (including those watching after matches the grand total of streams was 66.m); and visitors to the World Cup content on the BBC Sport website went up from 32.3m UK unique browsers in 2014 to 49.2m this year. If the same growth of online viewers happens again for 2022 then we could be looking at maybe 200m total streams.

Then for specific matches – first it was the BBC at 3.22million, then ITV pipped it to 3.3 million before the BBC pushed it up to 3.8million with the Quarter Final.. And, ITV then reached 4.3 million for the Semi Final. The final was a bit lower down at 2 million...

So, this leaves us with a new challenge to solve – how to test at this “real world” hyper-scale in the future? How do we scale it up too – 4x more streams in four years time? These traffic levels are not easy to emulate - for example: the traffic level to support the viewers for a live stream of a specific match is actually a multiple more than the UK Internet exchange LINX typically handles at peak time (which is around 4Tbps). Then start to factor in a switch to 4K (UHD) viewing and each of those streams uses 4x the bandwidth of a traditional HD stream!

If you think about “classic” performance testing methods - any set of automated performance tests will have a certain amount of flaws in it as the real world load patterns just isn’t that predictable – yes, you can build models and all sorts of probability based test suites to create to cover a whole variety of tests but, rarely can these truely emulate production user activities. Nor can these easily be done at actual production level scale (finding load injectors to emulate 4 million active end points is not easy) and if they were reproducible the repeated nature of the test models can flatten out impacts and effects you really want to see. Then, as often happens you scale down the test volumes to a more achievable size and scale via applying mathematical models this can often produces incorrect results as its actually very difficult to scale everything down equally as that’s one of the things about technology – most components are not that linear in its performance characteristics, sometimes they also can just fall off a cliff at a certain scale point too.

As we move forwards the growth seen in live IP based streaming of events will only continue – so there are lots of answers we do need to find... If you want to be part of cracking this one – I’d love you to join the team!