uncharted backwaters

Microsoft doesn't understand DNS

2014-07-02T12:29:59+00:00

A few days ago, Microsoft launched a DOS attack</a> against millions of users of the dynamic DNS service NO-IP</a>. Microsoft's aim was to disrupt DNS entries used by the creators of malware, but had the side effect of rendering the service effectively useless for anyone using DDNS for resolving servers, home automation systems, remote webcams etc.

Using a court order, Microsoft were able to replace the nameserver records for 22 domain names used by NO-IP and pointed them to their own nameservers. Microsoft's intent was presumably to answer legitimate DNS queries by forwarding them to the original NO-IP nameservers and block those from malicious entities, rather than completely crippling the service.

Microsoft have since claimed to have fixed the problem</a> yet for many users, myself included I can no longer find machines I use dynamic DNS to resolve. Many are attributing this to DNS propagation delay, but the actuality is that Microsoft seem to lack to lack the technical competence to implement a such a filtering system.

As of writing (02/07/2014 11:38:25 BST 2014) cached DNS entries for NO-IP domains are completely broken, so let's use dig to directly query the nameservers (and hence avoid any DNS caching issues):

$ dig -tNS +trace no-ip.org</span></span>
</span>
; <<>> DiG 9.9.2-P1 <<>> -tNS +trace no-ip.org</span></span>
;; global options: +cmd</span></span>
.                       163525  IN      NS      f.root-servers.net.</span></span>
.                       163525  IN      NS      a.root-servers.net.</span></span>
.                       163525  IN      NS      j.root-servers.net.</span></span>
.                       163525  IN      NS      k.root-servers.net.</span></span>
.                       163525  IN      NS      h.root-servers.net.</span></span>
.                       163525  IN      NS      e.root-servers.net.</span></span>
.                       163525  IN      NS      m.root-servers.net.</span></span>
.                       163525  IN      NS      b.root-servers.net.</span></span>
.                       163525  IN      NS      l.root-servers.net.</span></span>
.                       163525  IN      NS      i.root-servers.net.</span></span>
.                       163525  IN      NS      d.root-servers.net.</span></span>
.                       163525  IN      NS      g.root-servers.net.</span></span>
.                       163525  IN      NS      c.root-servers.net.</span></span>
.                       515689  IN      RRSIG   NS 8 0 518400 20140709000000 20140701230000 8230 . p5nawXXuH07BoGUsETH3J3VEj7W6H6V1EzwfRIRkr5qepcJQoqyGuced WTeOWW4kZV8GfB0NPS4Rp8HBfNTR6CsxNf4da92kTtbJKh9P+xEtreyd z3RDMqKDDBHGEl2Taml6J5Yhy89gsbigAZKPammqKh2aZM9+Tz46OmPt GHg=</span></span>
;; Received 913 bytes from 146.169.1.24#53(146.169.1.24) in 16 ms</span></span>
</span>
org.                    172800  IN      NS      a0.org.afilias-nst.info.</span></span>
org.                    172800  IN      NS      a2.org.afilias-nst.info.</span></span>
org.                    172800  IN      NS      b0.org.afilias-nst.org.</span></span>
org.                    172800  IN      NS      b2.org.afilias-nst.org.</span></span>
org.                    172800  IN      NS      c0.org.afilias-nst.info.</span></span>
org.                    172800  IN      NS      d0.org.afilias-nst.org.</span></span>
org.                    86400   IN      DS      21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2</span></span>
org.                    86400   IN      DS      21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0 D90F01BA</span></span>
org.                    86400   IN      RRSIG   DS 8 1 86400 20140709000000 20140701230000 8230 . F7mv0vQZqoDHVnIGnms53kiVT4nEwfxPv7ebMixzb20tI/FjH9nUrgvy PvrkzgXYV+HmO0Xzay/bsdLLhE2nMpFY7aXhbmzav9C126UGEAaheUDB 4MuTltNzmpu01biTVxyfqr6ZueE7QMWaKia/l29KdBab/3UgN7M3UL5e wr0=</span></span>
;; Received 683 bytes from 192.203.230.10#53(192.203.230.10) in 8 ms</span></span>
</span>
no-ip.org.              86400   IN      NS      ns7.microsoftinternetsafety.net.</span></span>
no-ip.org.              86400   IN      NS      ns8.microsoftinternetsafety.net.</span></span>
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN NSEC3 1 1 1 D399EAAB H9PARR669T6U8O1GSG9E1LMITK4DEM0T NS SOA RRSIG DNSKEY NSEC3PARAM</span></span>
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN RRSIG NSEC3 7 2 86400 20140723104131 20140702094131 21185 org. djVZn31Z2Fbpk8Wnj0QQ2HGfkZj/tU9UWhJIEViDbvPaKHfqHRVYnBLc 0n+s04e1uuZJpxmGOjIw6+aTJrxP/t4H525GtS5YLT/TeMQyK5Tq8dKN fUq0lpUqz0fhz2H8QhfHPpZDBCy1Udh29gPmAbXWb84yhEgra7jFObK5 VGA=</span></span>
vdarb7crtpe7mq8c176tsr178kc0put9.org. 86400 IN NSEC3 1 1 1 D399EAAB VDBA62E7405UOAVCP3IU953TNPO52T45 A RRSIG</span></span>
vdarb7crtpe7mq8c176tsr178kc0put9.org. 86400 IN RRSIG NSEC3 7 2 86400 20140722155512 20140701145512 21185 org. UAsAW27kIw8XUKxtz1nD4tsjF2uSf8ERmgZS02s4fb0ATIXbDY95Az+u 3Ai/iGDFjBxHJ1oJpEl8xat4IwHzx1/JEgVIheoOd6lklZbXFQRi7RAu E75yYUnnRyVCk1tqGBT3QpgPAM3U6dBkji0UZ/eAiL7CrTQk2Vzz3z8s cfY=</span></span>
;; Received 594 bytes from 199.19.54.1#53(199.19.54.1) in 156 ms</span></span>
</span>
no-ip.org.              119749  IN      NS      ns8.microsoftinternetsafety.net.</span></span>
no-ip.org.              119749  IN      NS      ns7.microsoftinternetsafety.net.</span></span>
;; Received 117 bytes from 157.56.78.73#53(157.56.78.73) in 142 ms</span></span></code></pre>
This shows that both the .org parent nameservers and the nameservers
Microsoft have introduced both agree that the current nameservers for
no-ip.org are located under microsoftinternetsafety.net. Their IPs
(157.56.78.93, 157.56.78.73) both belong to Microsoft. This means that
Microsoft's “fix” must involve changes to their nameservers and have not
returned nameserver control to NO-IP</em>.</p>
Let's lookup a DDNS host under the Microsoft nameserver, querying it
directly (replaced by asterisks for paranoia):</p>
$ dig -tA *****.no-ip.org @ns7.microsoftinternetsafety.net</span></span>
</span>
; <<>> DiG 9.9.2-P1 <<>> -tA *****.no-ip.org @ns7.microsoftinternetsafety.net</span></span>
;; global options: +cmd</span></span>
;; Got answer:</span></span>
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20400</span></span>
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1</span></span>
</span>
;; OPT PSEUDOSECTION:</span></span>
; EDNS: version: 0, flags:; udp: 4000</span></span>
;; QUESTION SECTION:</span></span>
;*****.no-ip.org.               IN      A</span></span>
</span>
;; ANSWER SECTION:</span></span>
*****.no-ip.org.        60      IN      A       31.53.88.84</span></span>
</span>
;; Query time: 217 msec</span></span>
;; SERVER: 157.56.78.73#53(157.56.78.73)</span></span>
;; WHEN: Wed Jul  2 11:49:05 2014</span></span>
;; MSG SIZE  rcvd: 60</span></span></code></pre>
It works, that's weird. This indicates that the Microsoft nameserver is
successfully forwarding requests to the underlying no-ip.org
nameservers. Now let's make the same request to Google's public DNS
server</a>:</p>
$ dig -tA *****.no-ip.org @8.8.8.8</span></span>
</span>
; <<>> DiG 9.9.2-P1 <<>> -tA *****.no-ip.org @8.8.8.8</span></span>
;; global options: +cmd</span></span>
;; Got answer:</span></span>
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 60346</span></span>
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1</span></span>
</span>
;; OPT PSEUDOSECTION:</span></span>
; EDNS: version: 0, flags:; udp: 512</span></span>
;; QUESTION SECTION:</span></span>
;*****.no-ip.org.               IN      A</span></span>
</span>
;; Query time: 297 msec</span></span>
;; SERVER: 8.8.8.8#53(8.8.8.8)</span></span>
;; WHEN: Wed Jul  2 11:52:09 2014</span></span>
;; MSG SIZE  rcvd: 44</span></span></code></pre>
This fails with an error suggesting that the nameserver Google contacted
failed to answer. Checking the no-ip.org nameserver entries show that
the data is up-to-date, which means that Microsoft's nameservers are
not replying to Google correctly.</p>
My first thought was that this was some weird geographical issue, or
maybe Microsoft had mistakenly identified Google's DNS requests as
malicious traffic. Instead the answer is much more simple, and
indicative of why Microsoft is simply incompetent.</p>
There are two types of DNS server: recursive and non-recursive.
Recursive DNS servers are typically contacted by end-users/clients, and
perform the sometimes complicated set of lookups required to resolve a
domain name. They almost always perform caching, in order to reduce load
on non-recursive nameservers.  Non-recursive name-servers serve only
local data, and don't perform DNS lookups themselves. Organisations
wishing to make DNS entries visible set up non-recursive DNS servers
so that they can be contacted by recursive ones.</p>
Within each DNS request, there is a “recursion desired”
flag</a> (RD). This
tells the DNS server whether or not it should recursively perform the
query. A recursive DNS server will perform requests with this bit
unset</em> since it's performing the recursion itself and the target DNS
server ideally shouldn't support it.</p>
Unfortunately Microsoft's DNS servers fail to deliver a useful reply if
this bit is unset. Consequently, djbdns</a>,
Google's nameservers</a>
and many other recursive DNS servers will never get a useful response
from the nameservers Microsoft have inserted.</p>
Let's perform the same lookup directly to Microsoft's nameservers again,
but this time disable recursion.</p>
$ dig -tA +norecurse *****.no-ip.org @ns7.microsoftinternetsafety.net</span></span>
</span>
; <<>> DiG 9.9.2-P1 <<>> -tA +norecurse *****.no-ip.org @ns7.microsoftinternetsafety.net</span></span>
;; global options: +cmd</span></span>
;; Got answer:</span></span>
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9751</span></span>
;; flags: qr ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 7</span></span>
</span>
;; OPT PSEUDOSECTION:</span></span>
; EDNS: version: 0, flags:; udp: 4000</span></span>
;; QUESTION SECTION:</span></span>
;*****.no-ip.org.               IN      A</span></span>
</span>
;; AUTHORITY SECTION:</span></span>
org.                    117492  IN      NS      c0.org.afilias-nst.info.</span></span>
org.                    117492  IN      NS      a0.org.afilias-nst.info.</span></span>
org.                    117492  IN      NS      a2.org.afilias-nst.info.</span></span>
org.                    117492  IN      NS      b0.org.afilias-nst.org.</span></span>
org.                    117492  IN      NS      b2.org.afilias-nst.org.</span></span>
org.                    117492  IN      NS      d0.org.afilias-nst.org.</span></span>
</span>
;; ADDITIONAL SECTION:</span></span>
c0.org.afilias-nst.info. 117492 IN      A       199.19.53.1</span></span>
a0.org.afilias-nst.info. 117492 IN      A       199.19.56.1</span></span>
a2.org.afilias-nst.info. 117492 IN      A       199.249.112.1</span></span>
b0.org.afilias-nst.org. 117492  IN      A       199.19.54.1</span></span>
b2.org.afilias-nst.org. 117492  IN      A       199.249.120.1</span></span>
d0.org.afilias-nst.org. 117492  IN      A       199.19.57.1</span></span>
</span>
;; Query time: 149 msec</span></span>
;; SERVER: 157.56.78.73#53(157.56.78.73)</span></span>
;; WHEN: Wed Jul  2 12:15:05 2014</span></span>
;; MSG SIZE  rcvd: 278</span></span></code></pre>
We do get a response, but it contains no useful information. This
demonstrates that Microsoft really has no idea what they're doing.</p>

ICFP Competition 2013 Writeup 2013-08-17T15:17:26+00:00 Introduction</h2> Having entered the International Conference on Functional Programming Competition for the last six years under the team name “Hacking in the Rain”, I thought it was about time I wrote about what I did. This year's competition</a> could effectively be summarised by the phrase “find the function” where the function mapped 64-bit integer values to other 64-bit integer values. In addition to the usual bitwise operators OR, AND, NOT, XOR, there was also addition, fixed offset shifts, conditionals and folds over all the bytes in a value with an arbitrary binary function. As always, I chose to implement in C++, not because I dislike functional languages, but because I'm somewhat more proficient in C++ than in Haskell, and I prefer the control I get when needing to solve efficiency-critical problems with limited computing resources. Overall Strategy</h2> The most obvious approach here would have been to try to enumerate all programs of increasing size until finding one that matched all the given input-output pairs, combined with some sort of pruning. I chose not to do this because I was afraid that constructing and evaluating every problem via this technique would be too slow, and never scale to the larger problem sizes. I didn't attempt to handle the “if0” or “fold” operators in my initial solver. My algorithm proceeded in two phases. In the first I computed metadata about constructable expressions and their sizes: Choose an input-output pair provided by the server. </li> Build a set containing only the values of expressions of size one. Namely, 1, 0 and the input value. </li> For increasing size, build sets containing all possible values of expressions of that size (excluding “if0” and “fold”). This only involves applying operators to elements of previously constructed sets and doesn't require any expression tree construction or evaluation. </li> Terminate when one of the constructed sets contains the output value we're looking for. </li> </ol> As this point, I know the size of an expression tree that computes the correct output for the input value I chose. However, I do not know what the expression actually is. The second phase builds the expression (or multiple expressions) that compute the output value from the input in a top-down manner. Given the desired output value and the size of the expression tree that computes it, I now find the tree itself: For every possible operator, determine if the value we want can be computed using the values we know we can compute from expressions with smaller sizes. We iterate over the sets we built in the previous phase to determine this. If yes, we know that this operator can form the topmost node in our tree. </li> We now need to find expression trees of for any operands used by the parent operator. We repeat step one for all subexpressions required by the parent. Again, we already know their size and value. </li> We terminate once we reach expressions of size 1. </li> </ol> Since there could be multiple expressions of the same size that evaluate to the same value (and I wanted to keep all of them) it was necessary to apply constructors across sets of values. I really missed being able to work in Haskell's list monad here which naturally handles Cartesian product-like constructions needed for the binary operator case. Armed with sets of expressions, it was a simple matter to evaluate them on all other input-output pairs I had requested from the server to validate. If no expressions turned out to be correct, I used the remaining input-output pairs to generate more expressions. This approach has the advantage that it only builds expression trees which are known to produce a correct answer for at least one input-output pair. In the event that no expressions were found that matched all input-output pairs, I added the option to deliberately construct oversized expressions. This was primarily used later for inferring conditionals for ifs. For many cases, this strategy allowed me to find correct expressions in less than a second, especially with operator restrictions. Handling “if0”</h2> Although “if0” could appear at arbitrary points in the expression tree, I only looked at generating expressions with “if0” at the top. As ifs (excluding those within folds) can always be lifted, this seemed practical. Though I could often find plausible expressions for sets of input-output pairs, none of these involved “if0” since trees were constructed for single input-output pair for which conditionals would always be constant. To solve this, I saved all programs that correctly evaluated a subset of the input-output pairs. Once I had at least one valid program for every input-output pair, I approximated a minimal covering</a> using a greedy algorithm to map each input-output pair to a correct program. I then used my existing code to infer the conditionals required to choose the correct program for each input-output pair. Depending on the number of programs used by the covering, this could involve a tree of nested “if0”s. This approach did work, however, it has a couple of subtle issues that were difficult to debug: Any input-output pairs that were correctly computed by more than one program used by the covering needed to be removed. Without doing this, it was possible for ambiguous pairs to be assigned the wrong program, and the if-condition would become impossible to derive. </li> The order of if-construction was important. As an example, assuming we were attempting to find this program: (lambda (id0) (if0 id0 1 id0))</code></pre> The following program (using _ as a placeholder) covers both sub-expressions, but it is extremely difficult to generate the condition without using another if: (lambda (id0) (if0 _ id0 1))</code></pre> This is a consequence of only having a bitwise NOT, and not a boolean NOT operator. To handle this I attempt to construct if conditions in all possible orders. </li> </ol> Results</h2> My final score was 701, which puts me at precisely 100th place. The unoffical scoreboard</a> gives a nice graphical representation of the problems I managed to solve</a>. Final Thoughts</h2> Although I never got around to handling folds, the strategy I implemented worked quite well for a number of problems. Far more than in other years, the lack of algebraic data types in C++ was painful to work around. I attribute this to both the number of possible node types in the tree, and the number of operations that needed to be applied to them. Of course, this is a good thing for a contest intended to advocate functional programming. I have mixed feelings over the decision to have the solver run on the client side, rather than by the competition organisers after the competition. I spent a lot of time trying to improve my solver before letting it loose, and only did when there were around 8 hours left before the competition ended. I suspect the scoreboard might look somewhat different if myself and others had had the time to apply their program to all the given problems. Updated 18/08/2013: Revised main algorithm description after feedback on reddit</a>. Another reason to ditch MythTV 2013-01-13T21:05:45+00:00 MythTV is very pretty and featureful, but has always struck me as a project that cares more about making it further so than trying to create a robust, bug-free core. Today I came across the sort of bug that exemplifies this. I decided to record the film “Unknown”</a>, starring Liam Neeson as a man who wakes from a coma to discover his life has been stolen. Here we have the film in the programme listing view: </a> Coincidentally, the MythTV front-end also lists time periods where it does not have any listings data as “Unknown”. The MythTV frontend rightly prevents the user from attempting to record these blocks. </a> Why is this an issue? It turns out that the MythTV front-end prevents you from recording any entry entitled “Unknown”, making it impossible to schedule recording of the aforementioned film. Clicking on the entry does nothing. Bringing up the menu does give you the option to record, but fails to schedule it. </a> The only way I found to schedule the recording was to do it from the HTML interface. Bad news for non-computer savvy Liam Neeson fans. Just for the record, the front-end version was a 13/01/2013 build of the 0.26-fixes branch. Updated 14/01/2013: Someone else apparently noticed around the same time</a>. The Amazon Kindle: wonderful device, shame about the books 2011-07-02T20:08:49+00:00 The only time I tend to have to read is when I'm travelling to and from work. Unfortunately, it's rather inconvenient for me to have to remove and replace the book from and to my rucksack. This sometimes also leads to me damaging the book in question. Hence, my decision to purchase an Amazon Kindle. The Kindle is certainly a nice device. It's reasonably light, and despite having a resolution of only 600x800, the properties of electronic ink and a pixel density of 167 ppi result in an impressive display. The user-interface can be a little clunky, but this is only an issue when you're doing something with the Kindle other than flipping the pages of a book. For my first read, I decided I wanted a complete collection of all Sherlock Holmes novels and short stories written by Sir Arthur Conan Doyle. Surely, not an unreasonable request since "The Adventures of Sherlock Holmes" (the first of four collections of stories) is prominently featured by Amazon as one of the free classics available for the Kindle. My first choice was to download them from Project Gutenberg</a> and avoid any DRM-related issues. Alas, none of these had any images (many stories were originally illustrated by Sidney Paget</a> when first published in The Strand magazine). Hence, I decided to purchase them from Amazon. I expected that it would be trivial to find a complete collection including original illustrations, ideally typeset for the Kindle, and not subject to typographical errors. I couldn't have been more wrong. Finding such a book in paper form isn't an issue. However, searching for an equivalent ebook led me to reviews complaining about missing images, missing contents pages, missing lines, poorly typeset conversations, poorly scanned images and adverts for other books embedded within the text1</a>. Note that most of these editions were non-free and it would be necessary to buy the ebook before spotting the issues, and again to get a revised version (if one was produced in the future). The first problem is that searching Amazon for Sherlock Holmes ebooks reveals many, many, results and it's extremely difficult to distinguish between them. Amazon's site aggregates the reviews of all editions of a book together, regardless of publisher. As a result, it's rather difficult to find reviews for a specific Kindle edition of a book. Multiple Amazon customers seemed confused</a> by which ebook edition was being described by which review. Unfortunately, if the quality of ebooks varies massively across editions, finding reviews for a specific edition becomes altogether more important. The next problem is the lack of trustworthy publishers. I had hoped that I might be able to make a guess about the quality of the ebook based on its publisher. Publishers of physical books tend to have well defined reputations but typing many of the ebook publishers into a search engine revealed no other sign of their existence. Other books were simply listed as public domain or had no publisher at all. One publisher's</a> website had vanished entirely. Of almost all the publishers I could find, there was no indication that they'd existed for more than a year or two. This is perhaps, hardly surprising. In the UK, all the Sherlock Holmes novels and short stories have passed into the public domain. It doesn't take much work to imagine a business model to become an ebook publisher selling only classics. Download out of copyright works from Project Gutenberg or other public domain resources.</li> Edit away undesired text and modify typography.</li> Create persuasive description and cover page, then sell on Amazon.</li> </ul> The best part is that creating an ebook has no physical printing cost. Of course, I have no way to know whether this is happening. What I do know is that the reviews suggest some extremely shoddy publishers. Given the difficulty of checking for issues before purchase and no way to return or get refunded for an ebook, it's probably rather easy to make a pretty penny. However, there is spark of hope. An illustrated edition</a> by Seashell Press</a> seems to have had an unparalleled amount of effort placed into its creation and received excellent reviews. It's probably also worth noting that this is the only publisher I've seen that also appears to produce physical books. Alas, it looks like this collection was originally complete, but then was revised to not contain "The Case-book of Sherlock Holmes" as it was still under copyright in the US. This change was also applied to the version available in the UK as well. I've since emailed the publishers to see if it is possible to get the original version distributed in the UK, where copyright of that particular collection is not an issue. Amazon's use of DRM, remote content removal</a> and the choice not to support EPUB</a> indicate how much they wish to retain control of Kindle ebook distribution. As such, the quality of ebooks available from the Kindle store directly affects the utility of the Kindle, and it's sad to see issues like these that I never anticipated. These are simply first impressions, and I have no idea if the issues I've described affect other books. If they do, they probably only exist for for older, out of copyright works. However, one of Amazon's main selling points for the Kindle seems to be that it provides an acceptable experience for reading classic literature. Just look at the Kindle's display every time it's switched off. ^1For example, this free edition</a> of The Adventures of Sherlock Holmes, currently at #3 in the Free Kindle store, contains character encoding errors. One such error is in the story "Adventure II. The Red-Headed League" which contains the text "there is now another vacancy open which entitles a member of the League to a salary of ï¿½4 a week for purely nominal services". The three strange characters should instead be a pound sign. In this non-free edition</a> of "The Complete Sherlock Holmes Collection", I found that the links in the table of contents for the stories "The Adventure of a Case of Identity" and "The Adventure of the Read-Headed League" pointed to each other's stories instead of their own. It seems free of more major issues though. </div> Random Slowness with Western Digital Caviar Green Hard Drives 2010-10-14T02:28:55+00:00 Several months ago, my MythTV</a> server started acting strangely. At seemingly random intervals, the system would experience high IO, completely crippling the responsiveness of the system to the extent that it would take minutes to even establish an SSH connection to the machine. There appeared to be no way to trigger the problem and running my own high-IO tasks showed no issues. The high-IO periods lasted for at least several minutes but sometimes occurred days apart, and as they wrecked the machine's responsiveness, it was also practically impossible to monitor the system when the symptoms did appear. Over the months, I tried changing the root file-system, searching for known IO issues with the Linux kernel, XFS or MythTV, checking fragmentation, swap-utilisation, IO-priority settings and running a multitude of different kernels including low-latency versions, all to no avail. The times I managed to SSH into the machine during a high-IO attack, running top and latencytop were equally unrevealing. It was only recently that I finally discovered the culprit: the Western Digital Caviar Green 1.5TB hard drive. It appears that the drive randomly enters periods where IO radically slows for periods of minutes, then recovers and behaves normally until the next one. There are no other signs that anything else is wrong with the drive such as strange noises, bad sectors or SMART warnings. The drive model in question is a 'WDC WD15EADS-00P8B0' and is only just over a year old which means the symptoms must have started manifesting not long after purchase. As the problem isn't reproducible on demand, I can't give solid IO performance figures. What I can state is that when the problem has manifested at boot time, I've given up waiting after tens of minutes for a system that usually boots in less that one. I can't imagine that extensive amounts of data are read during boot, which makes me wonder if the problem might be due to seeking rather than actual read speeds. Either way, the result is an unusable system. I've found postings on these issues here</a>, here</a> and here</a> which indicate the issues affect only specific models. As far as I can tell, Western Digital don't seem to have acknowledged any issues with this drive, nor provided any sort of firmware update that might fix it. As the symptoms don't match conventional signs of drive-failure (it's unclear if the problem is even physical), I can only imagine how many others are experiencing these issues without realising the cause. I'm not going to condemn Western Digital drives outright. The 2.5" 'WDC WD3200BEVT-35ZCT1' SATA drive in my laptop has been absolutely fine and is also the first laptop hard-drive I've owned that is so quiet that I actually need to look at the activity LED to judge IO load. However, I find it disappointing that Western Digital seems to have done so little to acknowledge what is clearly an issue affecting a number of people. Now let's see if I can get Western Digital to give me a RMA code. Debian OpenSSL Vulnerability Still Pains Two Years On 2010-08-17T17:58:18+00:00 In April 2006, someone filed a bug report</a> with Debian complaining that OpenSSL</a> (an open source SSL/TLS implementation) read data from an uninitialised buffer. This was causing Valgrind</a> (a brilliant debugging tool) to report memory usage warnings anywhere the OpenSSL random number generator was used. This was actually the behaviour intended by the OpenSSL developers, as the uninitialised buffer was being fed into an entropy pool used by the random number generator. A fix was proposed that reset the uninitialised buffer to zero before first use. This would have had minimal security implications as OpenSSL used multiple reliable entropy sources in addition to the buffer. However, the initial patch to do this didn't stop all the Valgrind warnings. Somehow the proposed fix mutated into one</a> that caused almost no entropy to be added to the pool except for the process ID, rendering the random number generator almost entirely useless. This in turn led OpenSSL to create extremely vulnerable SSL certificates, SSH keys and a bunch of other things. The broken OpenSSL version made it into Debian in September 2006. It later propagated into Ubuntu. Incredibly, no one noticed the vulnerability</a> until 2008. By then, a massive number of packages</a> and users had been affected. Cue scrambling to fix the bug in Debian and Ubuntu, creation of blacklists and scanners</a> for vulnerable OpenSSL and OpenSSH keys, arguments about who was</a> to</a> blame</a> and how open source practices could have failed so badly. Also</a>, cartoons</a>. Yes, I did feel the need to reiterate the entire bug's history. Reading the bug report is like watching a slow-motion car crash. Even the "fix" for the initial bug was applied incorrectly, which meant it appeared in Debian version 0.9.9c-1 of OpenSSL (released 17 September) instead of 0.9.8b-1 (released 4 May). I suppose that was a good thing. Even now, over two years after the bug was discovered and almost four years after it was originally introduced, the damage it caused is still being discovered. The Electronic Frontier Foundation</a>, launching a project</a> into all publicly used SSL certificates has had to delay because they are disclosing vulnerabilities to websites they found using weak private keys generated by the broken Debian/Ubuntu OpenSSL versions. As the EFF SSL Observatory page will be updated at some point, it currently reads: This project is not fully launched yet, because we are currently engaging in vulnerability disclosure for around 28,000 websites that we observed to be using extremely weak private keys generated by a buggy version of OpenSSL</a>. </blockquote> Further information can be found in the SSL Observatory DEFCON 18 slides</a>. Of the 28K vulnerable certificates seen, the 530 validating ones are the most interesting. The others were either invalid or the 12K issued by private certificate authorities. Only 73 of the 530 valid certificates had been revoked. In particular none of the 140 valid certificates by Equifax had been revoked, and only 4 of the 125 issued by Cybertrust. In conclusion, this all seems rather depressing. A bug in a patch to an important open source security library went unnoticed for two years, despite reducing of the effective security of the keys it generated to almost nothing (15 bits). Furthermore, a bunch of private "trusted" companies still haven't taken measures to ensure SSL certificates they generated using this library have been marked invalid another two years after the bug was found. On a positive note, I'm sure Debian has learnt from its mistakes by now, but it would still be nice to find some policy document to show what has changed (I failed to find one). Rather unsettlingly it seems like this was the exact type of bug the Debian Security Audit Project</a> was set up to spot. Unfortunately, nothing so positive can be said about the state of Certificate Authorities. Until one is sued for not taking proper security precautions, their behavior is unlikely to change. As for security, there's nothing to stop a CA from collaborating with a government or other entity that wants to eavesdrop on communications (as the EFF warns</a>). Also, no amount of money spent on an SSL certificate from even the most trustworthy CA will protect against a rogue certificate created (or successfully forged) from a different CA also trusted by the web browser. In other words, SSL is nowhere near as trustworthy as you think. Debian Packaging for TrueCrypt 2009-11-20T23:45:50+00:00 I like to be able to build Debian packages from source. Unfortunately, the released TrueCrypt</a> sources don't contain the Debian/Ubuntu packaging used to build their .deb files. As a result, I've created my own Debian packaging, available here</a>. As the TrueCrypt license seems to have issues that stop TrueCrypt being included in most distributions, the downloadable files only include the 'debian' folder. You'll still need to download the TrueCrypt sources from here</a>. Advantages over the upstream packaging include an init script to kill TrueCrypt device mappings on shutdown and the generation of a reasonably well formatted man page from the output of 'truecrypt --help'. DNS resolution delays in Debian (and Ubuntu) 2009-05-23T16:38:52+00:00 Yesterday, I was pinging a server when I noticed that the output of ping seemed to be rather slow. In fact, I'd noticed it before but never really thought about it until I was pinging anther server at the same time and the saw the drastic difference in output speeds. Pinging google.co.uk, there was a ping every second: fpr@callisto:~$ ping google.co.uk | perl -ne 'use Time::Format; print "$time{\"hh:mm:ss.mmm\"} - $_"' 14:44:34.898 - PING google.co.uk (74.125.77.104) 56(84) bytes of data. 14:44:34.915 - 64 bytes from ew-in-f104.google.com (74.125.77.104): icmp_seq=1 ttl=238 time=32.9 ms 14:44:35.883 - 64 bytes from ew-in-f104.google.com (74.125.77.104): icmp_seq=2 ttl=238 time=35.9 ms 14:44:36.882 - 64 bytes from ew-in-f104.google.com (74.125.77.104): icmp_seq=3 ttl=238 time=33.4 ms</code></pre> On another server, it was closer to five: fpr@callisto:~$ ping server1.fsckvps.com | perl -ne 'use Time::Format; print "$time{\"hh:mm:ss.mmm\"} - $_"' 14:49:42.389 - PING server1.fsckvps.com (66.71.248.146) 56(84) bytes of data. 14:49:42.408 - 64 bytes from 66.71.248.146: icmp_seq=1 ttl=45 time=122 ms 14:49:47.500 - 64 bytes from 66.71.248.146: icmp_seq=2 ttl=45 time=123 ms 14:49:52.625 - 64 bytes from 66.71.248.146: icmp_seq=3 ttl=45 time=122 ms</code></pre> This didn't make much sense since the round-trip times were small by comparison and ping sends one request per second by default. A Google search indicated that the problem might lie with my resolv.conf</a> file. Unfortunately, mine seemed to be fine and my local DNS server was completely responsive. However, if I pinged the server by IP address instead of by hostname, the delay was gone. To see what was blocking, I ran strace</a> on ping. 15:04:26.789 - munmap(0x7f69ed319000, 129482) = 0 15:04:26.790 - socket(PF_FILE, SOCK_STREAM, 0) = 4 15:04:26.790 - fcntl(4, F_GETFD) = 0 15:04:26.790 - fcntl(4, F_SETFD, FD_CLOEXEC) = 0 15:04:26.790 - connect(4, {sa_family=AF_FILE, path="/var/run/avahi-daemon/socket"...}, 110) = 0 15:04:26.790 - fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) 15:04:26.790 - fstat(4, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 15:04:26.790 - mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f69ed359000 15:04:26.791 - lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) 15:04:26.791 - write(4, "RESOLVE-ADDRESS 66.71.248.146\n"..., 30) = 30 15:04:31.789 - read(4, "-15 Timeout reached\n"..., 4096) = 20 15:04:31.789 - close(4) = 0</code></pre> It was blocking each ping on a read from /var/run/avahi-daemon/socket, a socket used by Avahi</a>, an implementation of the network auto-configuration and service discovery mechanism Zeroconf</a> that augments DNS. Killing the Avahi daemon solved the problem, but I still wanted to work out why ping was talking to Avahi and why it only occurred on certain hosts, so I ran ltrace</a>: 15:14:33.062 - gettimeofday(0x7fff90ec0550, NULL) = 0 15:14:33.062 - gettimeofday(0x7fff90ec0520, NULL) = 0 15:14:33.062 - memcpy(0x00608988, "\311\004\030J", 16) = 0x00608988 15:14:33.062 - sendmsg(3, 0x6077c0, 2048, 2, 61091) = 64 15:14:33.184 - recvmsg(3, 0x7fff90ec15f0, 0, 0, 61091) = 84 15:14:38.193 - gethostbyaddr("BG\370\222T\315(\002", 4, 2) = NULL 15:14:38.193 - inet_ntoa(0x92f84742) = "66.71.248.146" 15:14:38.194 - strcpy(0x006078e0, "66.71.248.146") = 0x006078e0</code></pre> I then wrote a little test in C just to check that I could replicate the delay with gethostbyaddr()</a>. I could, and it was then that I finally realised that the delay occurred when pinging hosts that had no PTR record (reverse DNS). Slightly confusingly, ping performs a reverse DNS lookup on the IP address when provided with a hostname, but not when given an IP address. gethostbyaddr() was calling Avahi because it had plugged itself into the glibc resolver using NSS</a>. When an attempt to resolve an IP address to a hostname failed, glibc would then call Avahi to try to find it. For whatever reason, Avahi cannot answer this request instantly and times out after 5 long seconds. Avahi also resolves host names to IP addresses but the delay in looking up unresolvable host names only occurs if the domain is under the .local pseudo top-level domain</a>. The Debian package dependencies make it a bit difficult to remove Avahi so the easiest way to fix this is to remove references to mdns4 from /etc/nsswitch.conf</a>. If you want to kill the daemon entirely then you can always disable its init script of course. Amusingly (or tragically) this bug is listed in Ubuntu</a> and Debian</a> bug reports which are both over two years old. At time time of writing, it still exists in Ubuntu Jaunty and Debian testing. What makes me really angry is that somewhere, someone decided that it would be a great idea to enable this daemon by default on desktop installs and as a result, performance of applications is being degraded. Obviously ping doesn't matter that much, but as mentioned in the bug reports, this hits people using ssh and IMAP as well, causing anything from mild delays to almost unusable systems. The worst part of this is that many people (and there could be a lot) suffering these issues are probably attributing them to a slow network, or packet loss, or ssh key verification or just about anything else other than a dubiously designed daemon running on their own machine. The fact that it only occurs on certain hosts only reinforces this and unless they suddenly realise that this delay is their local machine's fault and put a lot of effort into debugging, they'll probably never know. Transport for London, spam and all that 2009-01-22T01:51:40+00:00 I never realised this before, but Transport for London</a> have been spamming me for at least three years. Despite never having subscribed, I receive a weekly email</a> detailing weekend closures to various TfL services. I've happily ignored this abuse of my email address because unlike most spam, or more specifically in this case, Unsolicited Bulk Email, it came from a legitimate company, I knew how they got my email address (card application) and the messages contained genuinely useful information. Furthermore, I only want to unsubscribe as part of the process of moving to a new email address. Having said that, I'm not the only one who's noticed this abuse</a>. It turns out that getting off the mailing list is harder than it looks. The message contains no unsubscribe links. Neither does the TfL website. However, on the subscribe form, they do offer to stop duplicate emails if you give them your Oyster Card</a> number and your address. Well, that doesn't make me too paranoid about a giant government database. Examining the email's headers, they appear to be sent from the domain c-f-1.com</a>. Go to this domain and you'll get a PDF in which the company Communicator Corp</a> explain how they're not spammers and how every email sent though their system contains "an easy and automated way for you to unsubscribe" (they don't) and how they "promise to keep your email address secure and private" (also untrue). I'd have found a link to an unsubscribe form far more convincing. However, using Google, I was able to locate unsubscription forms</a> for other email sent from c-f-1.com. From this, I constructed a URL of the form: http://www.c-f-1.com/Unsubscribe.aspx?emailid={emailid} where {emailid} was the value of the 'X-UEmailID' header of the TfL message. Imagine my geekish glee when I was presented with an unsubscribe page containing my old email address. I still received an email the following week, but now it seems they've finally stopped sending messages to that address. What's rather disturbing is that those unsubscribe pages were even on Google in the first place. Looking at the email IDs, they appear to be nothing more than Base64</a> encoded integers. Whilst they don't seem to start from 0, they don't appear to be random either. Also, even after unsubscribing, these pages seem to remain. Yes, it seems Communicator Corp is leaking their their entire list of email addresses onto the World Wide Web. If that isn't a breach of privacy, not to mention the Data Protection Act</a>, I don't know what is. All that's needed now is for an enterprising spammer to write a script to collect them all. Update 26/04/2009: For the first couple of months after writing this, I noticed a number of hits to this page from what appeared to be Communicator Corp's Internet gateway. I can't be certain that it's connected, but I notice that the mailing list unsubscribe form now no longer leaks email address information and requires that the email address be entered as confirmation which is a definite improvement. If Communicator Corp are reading this, I'd also like to take this opportunity to ask them to ensure that all messages sent do actually have a link to the corresponding unsubscribe form. Update 02/10/2009: I notice that the weekly TfL emails now have both subscribe and unsubscribe links although I'm unsure when they appeared. MD5 collision used to create rogue certificate authority 2009-01-01T13:48:04+00:00 This</a> really is quite nice. Researchers used collisions in the MD5 hash algorithm to create a rogue CA (Certification Authority) certificate signed by RapidSSL</a>. RapidSSL is apparently trusted by the majority of web browsers. It was demonstrated</a> years ago that MD5 collisions could be generated relatively easily, but that hasn't stopped MD5 being used in a number of contexts where cryptographically secure hash functions are required. The attack involves getting a CA that uses MD5 for hashing to provide a legitimate website certificate. The attacker then generates their own CA certificate which has the same hash as the legitimate one. It is now possible for the attacker to generate SSL certificates for arbitrary websites, signed by their rogue CA certificate. As the rogue CA certificate has the same hash as a legitimate one signed by the CA, browsers that trust the CA will accept the fraudulent site's identity. What I really like about this work is that the authors have managed to bridge the gap between a result primarily of interest to security researchers and an issue which could affect the average web user. Quite often, when cryptographic research results get publicity, the implications are so obscure to anyone without a knowledge of cryptography that the reporting soon becomes badly distorted. Hopefully, this example of how MD5's unsuitability as a cryptographic hash can lead to such an easily comprehensible real-world vulnerability might make people take notice of the danger in using broken hashing and encryption algorithms. Well, one can hope. As Bruce Schneier points out</a>, the plethora of valid sites with broken certificates have trained users to ignore SSL warnings so the ability to spoof SSL certificates doesn't really add that much. Having said that, Firefox 3 goes out of its way to make it difficult to access web-sites with invalid SSL certificates. Still, given the multitude of far simpler methods criminals have for acquiring sensitive information, it's unlikely this attack will ever be seen in the wild.

uncharted backwaters

Microsoft doesn't understand DNS

ICFP Competition 2013 Writeup

Introduction</h2> Having entered the International Conference on Functional Programming Competition for the last six years under the team name “Hacking in the Rain”, I thought it was about time I wrote about what I did.</p>

Results</h2> My final score was 701, which puts me at precisely 100th place. The unoffical scoreboard</a> gives a nice graphical representation of the problems I managed to solve</a>.</p>

Another reason to ditch MythTV

The Amazon Kindle: wonderful device, shame about the books

Random Slowness with Western Digital Caviar Green Hard Drives

Debian OpenSSL Vulnerability Still Pains Two Years On

Debian Packaging for TrueCrypt

DNS resolution delays in Debian (and Ubuntu)

Transport for London, spam and all that

MD5 collision used to create rogue certificate authority

Results</h2>
My final score was 701, which puts me at precisely 100th place. The unoffical scoreboard</a> gives a nice graphical representation of the problems I managed to solve</a>.</p>