Web Server Stress Testing with Curl

I use OpenSTA a lot for stress testing but it can take a bit of time preparing tests. We have a product that has been causing some problems in production with some rare exception conditions. What I wanted was a list of all the URLs used to access the website and then cycle randomly over this list with the option of testing with multiple users. Now I could extract the URLs from the web server log files or use a URL spider to obtain a complete list. These could then be wrapped into an OpenSTA test script. However after I spidered the site I had a list of 45,000 URLs. That seemed a bit much for OpenSTA. Instead I wrote a shell script using Curl. This logs the results in CLF format so it can be analyzed, although probably with AWK rather than Excel given the quantity of the data.

The script can test multiple users and perform multiple iteration or stop after a given time. Results are logged to a file.

# kill all subshell processes
killall() {
	kill $(ps | grep '/sh$' | tr -s ' \t' ',' | cut -d',' -f 2) > /dev/null 2>&1
	exit
}
 
trap killall INT TERM EXIT
# check we have the number of users, iterations and output file for results
if [[ $# -ne 4 ]]
then
	echo "Usage: $0 <users to run> <iterations> <time in seconds: -1 for ever> <output file>"
	exit 1
fi
users=$1
iterations=$2
time=$3
file=$4
 
if [[ $time -eq -1 ]]
then
	echo "running with $user user for $iterations iterations"
else
	echo "running with $user user for $iterations iterations and maximum time $time seconds"
fi
 
# start users as sub processes
echo -n "started "  > $file
date >> $file
for ((j=1;j<=$iterations;j++))
do
	for ((i=1;i<=$users;i++))
	do
		echo "starting user $i"
		(
		echo $PID
		shuf urllist.txt | while read url
		do
		curl -s --max-time 30 -e "EzStress Tool User $i" -o /dev/null -w @logformat.txt --url $url >> $file
		done
		)&
	done
done
if [[ $time -eq -1 ]]
then
	wait
else
	sleep $time
	killall
fi
echo -n "done " >> $file
date >> $file

The important thing is that for each subthread we randomize the set of URLs to be tested. The file logformat.txt says what information will get logged

%{url_effective},%{http_code},%{content_type},%{time_total},%{time_connect},%{time_starttransfer},%{size_download}\n

See the Curl documentation for more information.