I’d like to have gone into this whole thing in a bit more of a steady manner, but as luck would have it, I’ve been thrown into the proverbial deep-end by happenstance. I’m currently part of a task force who will be attacking the concept of Load Testing for our company. We’ve had some shaky days with massive influxes of traffic in the past, so we’re aiming to go into this thing fully prepared for every eventuality.
First stage was selecting an appropriate software. There are lots of them around, so we set about defining a criteria for what we regarded to be ‘must-have’ features. For us, this criteria was as follows:
- Can be integrated into our CI/CD pipeline
- Easy to measure our system during a load test
- Good reporting of results
- Ability to write complex scenarios
- Ability to version control the tests
Since we wanted to version control the tests and integrate them into our delivery pipeline, this ruled out pretty much all SaaS Load Testing products. Which on some level, is a shame, since they offer some excellent reporting tools.
We reviewed a number of different platforms and the one we settled on was Locust. These guys have done a great job in making the codebase very easy to use. It’s all written in Python, and as such, you have access to a wealth of Python-based tooling, making the whole thing really flexible.
Installing Locust
This is where we had the most trouble. Getting locust installed was a bit of a job – I’ll preface this by saying that the issues we experienced at this stage, have now been resolved by locust from version 0.7.5 onwards.
We discovered that Locust didn’t support SSL. This wasn’t Locust’s fault – a dependency’s dependency had dropped some essential, breaking code into a minor version release. All guides we found suggested that this was due to Python dropping SSL support in their version too, so the recommended course of action was to rollback Python to a version < 2.7.8.
We used pyenv and virtualenv to try and get this to roll back effectively, and then other things began to break. It wasn’t easy. Then I discovered that Locust’s dependency gevent had resolved this issue in their latest version, so I had to wipe references to pyenv and virtualenv reinstall python ( brew install python ) and upgraded gevent directly ( pip install gevent==1.1.1 ).
Everything began working smoothly.
The good news is that Locust engineers have now resolved this issue and are now packaging gevent 1.1.1 with the installer, so you’ll likely never encounter this issue.
Writing Tests
One of the biggest challenges I faced with Locust was discovering how to structure our project. Locust provides you with a set of tools and instruction on how to use them, but if you’re doing anything more substantial than the simple examples they offer you, then you’re working in an essentially freeform project environment.
Throw into the mix that I’d never worked in Python before, and you have a recipe for confusion.
We eventually separated our tasks from our tests themselves and created Page objects to manage the route retrievals. It went through a number of iterations before we decided on a solution.
Once that part was done, we realised that much of the hardest work was over and we could begin writing our tests in earnest.
Running The Tests
We opted for a series of Digital Ocean droplets so we could distribute the tests across a number of data centres. This approach allowed us to hit our staging server with a large amount of virtual users from different locations.
We used New Relic, Dataloop and Grafana to see in real time what the tests were doing to our system. We could see what the slowest database queries were, we could identify bottlenecks in the code, and we could track processing timings over time. We made some extremely interesting discoveries.
From this, we were able to make targeted hits of refactoring in the system to really focus down on the areas that weren’t fit for purpose. Before we knew it, we had worked out all the kinks and issues with the system and we were able to max our webserver’s CPU, forcing an auto-scale to continue serving content with negligible issues.
writer • coder • father • husband 