Does the TFS Server have plenty of resources? I've never seen this, but I've seen something similar when the ASP_NET worker process is receycling very frequently due to memory etc being low on the TFS AT (usually when running the server in an underpowered virtualised environment)

Also, are you sure that your connection is not being proxied by some network device, firewall or NAT?