Two weeks ago my browser started acting weird, where weird means eating 99% CPU and never loading some pages. At first I ignored it, it just couldn't load some truecrypt documentation which I could check at work. But soon it got worse, more pages weren't loading so I decided to take some time and look into it.
It turns out pages that weren't loading were those either built with frames or containing iframes. A quick look at the javascript console revealed no errors, nor the page's source contained anything suspicious. Incidentally that day I wanted to rip some new CDs but ripperX would fail to look up info on freedb, which is done via a web request. A coincidence? My attention moved from the browser to the actual network traffic, and a dump with tcpdump showed quite a few surprising things. The first attracting my attention was the following header:

Via: 1.0 PSBTTEST:3131 (squid/2.6.STABLE6.2.7-6)

Ugh, a proxy! For several reasons I'm an aficionado of whatsmyip.org, which among other things is able to report if you are behind a proxy and give you some information about it by parsing additional headers like X-Forwarded-For. And by that I'm quite confident in saying that this proxy thing didn't exist up to the week before. The reported ip was 217.41.217.22, which a quick whois shows belonging to BT:

inetnum:      217.41.216.0 - 217.41.218.255
netname:      BT-MIDBAND
descr:        BtnMidband
country:      GB

Since BT is able to push updates onto customers' router via CPE CWMP (I'm leaving discussing the security of this solution to another article) I thought they might have tried to improve my internet experience... but I didn't find any differences with the configuration I saved last month. Having ruled out my system and the router, the only possible explanation was a transparent proxy, which as annoying as it can be is still very common and legit.
Being a regular on irc.freenode.net I thought of paying a visit to #squid and have a chat with those nice guys about transparent proxying technologies. And it proved to be useful as I learnt a couple interesting things. The first one is TRACE http method, which can be used in a traceroute fashion, where the Max-Forwards header plays the part of TTL. An example:

spike in space ~ > telnet www.henriknordstrom.net 80
Trying 81.233.163.21...
Connected to henriknordstrom.net.
Escape character is '^]'.
TRACE / HTTP/1.1
Host: www.henriknordstrom.net
Max-Forwards: 1

HTTP/1.0 200 OK
Date: Sat, 30 Jun 2007 22:59:44 GMT
Server: Apache/2.0.40 (Red Hat Linux)
Content-Type: message/http
Via: 1.0 PSBTTEST:3131 (squid/2.6.STABLE6.2.7-6)
Connection: close

TRACE / HTTP/1.0
Host: www.henriknordstrom.net
Max-Forwards: 0
Via: 1.1 PSBTTEST:3131 (squid/2.6.STABLE6.2.7-6)
X-Forwarded-For: 86.130.248.224
Cache-Control: max-age=259200
Connection: keep-alive

Connection closed by foreign host.

The other thing I learnt is all versions of squid prior to 2.6.12 suffer from a couple serious security vulnerabilities...
But so long nothing new, big telcos running bugged software is unfortunately quite a common practice.
But I was just scratching the surface. Another look at the dump revealed several referrals to dns.sysip.net (212.187.177.142), an ip owned by Level3:

inetnum:        212.187.177.136 - 212.187.177.143
netname:        INTEGRATED-COMMUNICATIONS-UK
descr:          BBBG58761 vlan 396
country:        GB
admin-c:        LTHM
tech-c:         LTHM

A curious thing about this ip is the hostname it resolves to, unknown.Level3.net. Nothing wrong with funny names, we use tons where I work, but still, "unknown" doesnt exactly sound good to me. What sounds even worse is the first page of results you get when googling for dns.sysip.net. Twenty out of twenty links I visited were mentioning it as part of spyware and suggesting various windows software to clean browsers' cache, registry and so on. Now that's a piece of news!

About the spyware

From a network perspective any website I'd connect to, included spikelab.org, would result in an initial GET to the original site immediately followed by a GET to dns.sysip.net, which would download some html/js. Testing against mp3.com:

<html>
  <head>
    <noscript><meta http-equiv="Refresh" content="0;
URL=http://dns.sysip.net/services/nojscript?orig=http%3A%2F%2Fwww.mp3.com%2F&request-id=1183416188_268148"/></noscript>
<script type="text/javascript"
src="http://dns.sysip.net/services/bind?request-id=1183416188_268148"></script>
    <script type="text/javascript">
    try {
      var domain = location.hostname, cookie_str = "ps-uid=" + UID;
      if (!domain.match(/^(\d{1,3}\.){3}(\d{1,3})$/)) {
        var h = domain.split("."), l = h.length;
        if (l >= 2) {
          domain = "." + h[l - 2] + "." + h[l - 1];
          if (l >= 3 && (h[l - 2] == "com" || h[l - 2].length < 3))
            domain = "." + h[l - 3] + domain;
        }
        cookie_str += "; domain=" + domain;
      }

      var d = new Date();
      if (UID != "OPTED_OUT") {
        d.setTime(d.getTime() + 1000 * 60 * 60 * 24 * 3);
      } else {
        // set to 4 hours
        d.setTime(d.getTime() + 1000 * 60 * 60 * 4);
      }

      cookie_str += "; path=/; expires=" + d.toGMTString();
      document.coo
kie = cookie_str;
    } catch(e) { }
    window.location = "http://www.mp3.com/";
    </script>
  </head>
  <noscript><iframe
src="http://dns.sysip.net/services/frame?orig=http%3A%2F%2Fwww.mp3.com%2F&request-id=1183416188_268148"
width="1px" height="1px" style="visibility:hidden;">
</iframe></noscript>
</html>

That triggers a http request from the client to dns.sysip.net via the iframe, which results into this:

HTTP/1.0 200 OK
Set-Cookie: uid=Khwlm+0o3BGy4AAUXtpNIA||; expires=Thu, 5-Jul-2007 22:43:08 GMT; path=/services/
p3p: CP="NOI DSP LAW CURa DEVa TAIa PSAa PSDa OUR STP BUS UNI COM NAV INT"
Date: Mon, 2-Jul-2007 22:43:08 GMT
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/x-javascript
Connection: close

var UID="Khwlm+0o3BGy4AAUXtpNIA||";

My best guess is they are logging the request plus setting a variable and cookie for tracking purposes. Calling that url manually and fiddling with the request-id parameter results in a similar value in output with the only first 8 chars changing, which suggests some kind of weak encryption schema where the second part is based on the client ip or similar.
On a website with no ads like mine that is the end of it. But on something riddled with ads like mp3.com the story goes on. Further down the dump we can observe:

206.16.21.32:www AP > space.home:57228 (1420)
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
p3p: CP="NOI DSP LAW CURa DEVa TAIa PSAa PSDa OUR STP BUS UNI COM NAV INT"
Date: Mon, 2-Jul-2007 22:43:23 GMT
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Set-Cookie: ps-origreferer=http://www.mp3.com/; path=/CNT/iview/gmsptcbr1380000276cnt/direct;wi.300;hi.250/01/2007.07.02.18.43.09
Set-Cookie: ps-request-id=1183416203_204989; path=/CNT/iview/gmsptcbr1380000276cnt/direct;wi.300;hi.250/01/2007.07.02.18.43.09

A1F
<html>
  <head>
    <noscript><meta http-equiv="Refresh" content="0;
URL=http://dns.sysip.net/services/nojscript?orig=http%3A%2F%2Fview.atdmt.com%2FCNT%2Fiview%2Fgmsptcbr1380000276cnt%2Fdirect%3Bwi.300%3Bhi.250%2F01%2F2007.07.02.18.43.09%3Fclick%3Dhttp%3A%252F%252Fadlog%252Ecom%252Ecom%252Fadlog%252Fe%252Fr%253D7166%2526s%253D740157%2526o%253D1%3A%2526h%253Dpi%2526p%253D%2526b%253D4%2526l%253DEn_US%2526site%253D6%2526pt%253D%2526nd%253D%2526pid%253D%2526cid%253D%2526pp%253D%2526e%253D%2526rqid%253D00c17-gne-ad5468237F130864CA%2526t%253D2007.07.02.18.43.09%2526event%253D58%252F&request-id=1183416203_204989"/></noscript>
      var domain = location.hostname, cookie_str = "ps-uid=" + UID;
      if (!dom
206.16.21.32:www A > space.home:57228 (1420)
ain.match(/^(\d{1,3}\.){3}(\d{1,3})$/)) {
        var h = domain.split("."), l = h.length;
        if (l >= 2) {
          domain = "." + h[l - 2] + "." + h[l - 1];
          if (l >= 3 && (h[l - 2] == "com" || h[l - 2].length < 3))
            domain = "." + h[l - 3] + domain;
        }
        cookie_str += "; domain=" + domain;
      }

      var d = new Date();
      if (UID != "OPTED_OUT") {
        d.setTime(d.getTime() + 1000 * 60 * 60 * 24 * 3);
      } else {
        // set to 4 hours
        d.setTime(d.getTime() + 1000 * 60 * 60 * 4);
      }

      cookie_str += "; path=/; expires=" + d.toGMTString();
      document.cookie = cookie_str;
    } catch(e) { }
    window.location =
"http://view.atdmt.com/CNT/iview/gmsptcbr1380000276cnt/direct;wi.300;hi.250/01/2007.07.02.18.43.09?click=http:%2F%2Fadlog%2Ecom%2Ecom%2Fadlog%2Fe%2Fr%3D7166%26s%3D740157%26o%3D1:%26h%3Dpi%26p%3D%26b%3D4%26l%3DEn_US%26site%3D6%26pt%3D%26nd%3D%26pid%3D%26cid%3D%26pp%3D%26e%3D%26rqid%3D00c17-gne-ad5468237F130864CA%26t%3D2007.07.02.18.43.09%26event%3D58%2F";
    </script>
  </head>
  <noscript><iframe src="http://dns.sysip.net/services/frame?orig=http%3A%2F%2Fview.atdmt.com%2FCNT%2Fiview%2Fgmsptcbr1380000276cnt%2Fdirect%3Bwi.300%3Bhi.250%2F01%2F2007.07.02.18.43.09%3Fclick%3Dhttp%3A%252F%252Fadlog%252Ecom%252Ecom%252Fadlog%252Fe%252Fr%253D7166%2526s%253D740157%2526o%253D1%3A%2526h%253Dpi%2526p%253D%2526b%253D4%2526l%2
space.home:57228 A > 206.16.21.32:www (0)
206.16.21.32:www APF > space.home:57228 (308)
14     FIN-WAIT-1     space.home:57228 > 206.16.21.32:www
53DEn_US%2526site%253D6%2526pt%253D%2526nd%253D%2526pid%253D%2526cid%253D%2526pp%253D%2526e%253D%2526rqid%253D00c17-gne-ad5468237F130864CA%2526t%253D2007.07.02.18.43.09%2526event%253D58%252F&request-id=1183416203_204989" width="1px" height="1px" style="visibility:hidden;"></iframe></noscript>
</html>
0

I'm not entirely sure what's happening here, the same technique is used, hidden iframe calling home with one of the ads link as parameter. It seems to do this for all the ads on the page, maybe in some sort of referral-based revenue schema. Here the triggered request:

Host: dns.sysip.net
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20061201 Firefox/2.0.0.3 (Ubuntu-feisty)
Accept: */*
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://view.atdmt.com/CNT/iview/gmsptcbr1380000276cnt/direct;wi.300;hi.250/01/2007.07.02.18.43.09?click=http:%2F%2Fadlog%2Ecom%2Ecom%2Fadlog%2Fe%2Fr%3D7166%26s%3D740157%26o%3D1:%26h%3Dpi%26p%3D%26b%3D4%26l%3DEn_US%26site%3D6%26pt%3D%26nd%3D%26pid%3D%26cid%3D%26pp%3D%26e%3D%26rqid%3D00c17-gne-ad5468237F130864CA%26t%3D2007.07.02.18.43.09%26event%3D58%2F
Cookie: uid=Khwlm+0o3BGy4AAUXtpNIA||

As you can see the cookie set during the first request is now used.
The other interesting bit is the ip involved in the request, 206.16.21.32 :

CERFnet CERFNET-BLK-206 (NET-206-16-0-0-1) 
                                  206.16.0.0 - 206.19.255.255
CERFnet customer - University of La Verne CERF-ULAVERNE (NET-206-16-20-0-1) 
                                  206.16.20.0 - 206.16.23.255

An university... how is an university involved with mp3.com and a spyware domain? The most likely answer to me is a cracked account used to bounce from, or something along those lines.
Another thing worth mentioning, that I believe confirms the role of the proxy, is what happens when you fetch the same page with curl. curl mp3.com | grep sysip gives no results, but add -A "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)" and you're served. I think that here BT is filtering for user-agent in order to save bandwidth and CPU by avoiding serving requests made by robots, which in turn means no spyware.

Two hypothesis

I hate uncertainty but with the data I can collect all I can do is formulating two hypothesis:
1) BT is pushing spyware onto its customers
2) BT's proxy has been cracked and used to infect customers
There's actually a third hypothesis based on what's been going on during the last 3 weeks with some ISPs injecting ads into pages by rewriting content on the fly. This doesnt seem the case of BT but they could have adopted some similar but less obvious technique.

An update

At the beginning of the article I mentioned that it all started two weeks ago. Unfortunately I've been away and too busy to produce this report sooner, and at the time of speaking the proxy is gone and there are no longer traces of sysip.net. Based on my dumps it seems the proxy was gone two days ago. Additionally something changed in the routes. Before a tcptraceroute would produce something along the lines of:

...
...
 5  217.41.171.74  55.955 ms  48.530 ms  43.221 ms
 6  217.41.217.114  41.263 ms  41.745 ms  42.060 ms
 7  www.spikelab.org (217.147.94.42) [open]  48.709 ms  50.791 ms  61.070 ms

now the same command gives me:

...
...
 4  192.168.50.253 34.117 ms  38.298 ms  41.161 ms
 5  217.41.171.74  55.955 ms  48.530 ms  43.221 ms
 6  217.41.171.42  43.639 ms  46.307 ms  59.785 ms
...
...
13  rbii-rt2-e1.blackcat.net.uk (217.147.87.204)  40.021 ms  53.248 ms  48.008 ms
14  spikelab.org (217.147.94.42) [open]  56.927 ms  58.255 ms  46.341 ms

It's interesting to also notice the internal ip hop. Such node started showing up two days ago when the problem was "fixed", and during that day it would come up as last node, which I didn't really manage to explain, but which seems to be causing quite a few RST.

Conclusions

I have obviously a very limited insight of the situation, especially because all the data in my possession is limited to legit one dumped during normal usage, which doesnt offer any information about the status of BT's network. Nonetheless something happened, and being over or not, and I'd very much like to understand what it was. Anybody with more information or interest on the topic is welcome to email me. All dumps, chat logs etc are available on request. Thanks.