sender and replier need couples therapy

[rzezeski]

Both the README and code comments would have you believe that only the sender requires options. That you can simply start the replier and it will base its behavior (per connection) on the sender's options as passed to -o. This would lead one to believe that the sender and replier negotiate the benchmark terms before starting the benchmark. However, while trying to diagnose something else, I noticed some unexpected behavior from the replier process. For each message from the sender it used two recv(2) calls. This DTrace snippet below makes the behavior clear:

# dtrace -n 'syscall::recv:entry /cpu == 19/ { @["ask"] = quantize(arg2); @s[ustack()] = count(); }' -c 'psrset -e 5 /zones/826df1b9-e66f-493e-8bd2-8125a3b04bf3/root/root/seqtest-master/seqtest -S -o ssize=200,rsize=200,threads=1,count=100000 192.168.241.75,192.168.241.74:6667'
dtrace: description 'syscall::recv:entry ' matched 1 probe
ssz_max 200
Address 0: Host 192.168.241.74 Port 6667
t->ssz_max 200
Received 100000 replies
Time: 5121763.8 us
ROUND TRIP LATENCY:
Average:  49.5 us
Stddev:   0.8 us
Median:   49.5 us
90.0%ile: 49.9 us
99.0%ile: 50.3 us
99.9%ile: 52.7 us
Minimum:  42.7 us
Maximum:  127.9 us
dtrace: pid 23264 has exited

  ask
           value  ------------- Distribution ------------- count
              16 |                                         0
              32 |@@@@@@@@@@@@@@@@@@@@                     100001
              64 |                                         0
             128 |@@@@@@@@@@@@@@@@@@@@                     100000
             256 |                                         0


              libc.so.1`__so_recv+0xa
              libsocket.so.1`recv+0x22
              seqtest`replier+0xfb
              libc.so.1`_thrp_setup+0x8a
              libc.so.1`_lwp_start
           200001

Notice there are 200K calls to recv(2) on the replier alone. That's 100K more than expected.

If I pass options to the replier, so that it is aware of the proper size, things work as expected.

# psrset -e 6 /zones/826df1b9-e66f-493e-8bd2-8125a3b04bf3/root/root/seqtest-master/seqtest -o ssize=200,rsize=200 -r 192.168.241.74:6667
Address 0: Host 192.168.241.74 Port 6667

# dtrace -n 'syscall::recv:entry /cpu == 19/ { @["ask"] = quantize(arg2); @s[ustack()] = count(); }' -c 'psrset -e 5 /zones/826df1b9-e66f-493e-8bd2-8125a3b04bf3/root/root/seqtest-master/seqtest -S -o ssize=200,rsize=200,threads=1,count=100000 192.168.241.75,192.168.241.74:6667'
dtrace: description 'syscall::recv:entry ' matched 1 probe
Address 0: Host 192.168.241.74 Port 6667
Received 100000 replies
Time: 5121178.0 us
ROUND TRIP LATENCY:
Average:  49.5 us
Stddev:   0.7 us
Median:   49.5 us
90.0%ile: 49.9 us
99.0%ile: 50.4 us
99.9%ile: 52.0 us
Minimum:  38.3 us
Maximum:  126.6 us
dtrace: pid 23203 has exited

  ask
           value  ------------- Distribution ------------- count
              64 |                                         0
             128 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 100001
             256 |                                         0


              libc.so.1`__so_recv+0xa
              libsocket.so.1`recv+0x22
              seqtest`replier+0xfb
              libc.so.1`_thrp_setup+0x8a
              libc.so.1`_lwp_start
           100001

My preferred fix:

1. Implement the sender/replier negotiation.
2. For each benchmark connection have the replier print a one line summary describing negotiated parameters.
3. If someone passes -o to replier print usage and exit.

[gdamore]

Negotiation seems a little over kill. There already is some of the details that are negotiated -- the inbound packets contain a header that describes things; what they don't do is negotiate the frame size apriori.

Note that with randomization, the replier needs to be prepared to receive a message of unknown size. I've tried to make this work properly in the replier, where it fully buffers things, etc. but as you note, it will occasionally require more system calls than we'd like.

This can't really be solved without knowing apriori a fixed message size, because if you either have to read just a little bit for the header first (so 2 recv/read calls), or you have to read a large frame, which will work fine unless more data has arrived than a full frame, in which case you might wind up having a fragment in your buffer from the next frame.

At any rate, the intention here is not to measure the number of system calls, but to measure TCP performance & function.

I would go ahead and document an option to the replier to set the frame size, along with a recommendation that this be used when the sender is using a fixed frame size, to reduce the number of recv calls done on the replier.