Reverse Entropy

You can find many discussions on the web, and in books, about why you should use a certain design pattern in the architecture of your software.

Almost all of these discussions will propose adding some extra layers, and give one of the major benefits as being "this will decouple the X from the Y", with underlying assumption that "more abstraction is better".

This is obviously true. If you have two layers, A and B, which currently talk to each other, and then you add a layer C in between, now A and B don't need to talk to each other.

But all these earnest discussions neglect to consider the cost of doing this versus the benefit. If the cost outweighs the benefit, why would you do it, even if you like the benefit in isolation?

So what are the costs of extra layers in software?

Well, at the least they typically increase the work to add a new feature (where feature is defined as "a new concept that the application handles", not just a change to user interface). Instead of changing one place, you now have to touch N places. You also have to make sure that the interaction between them works correctly.

How about maintenance? Another oft-cited advantage of layers is that they will "make maintenance easier". This is often stated as an assumption with no evidence. In fact, my own experience and some research would suggest that often layers make maintenance harder. Tracking down bugs now means looking through multiple layers.

Of course one hopes that eventually, you'll find the bug, and it will be in one layer, and you fix that and there are no side effects. Again, this is too simplistic. Some bugs are like that, but often bugs are misunderstandings of the problem or bad design that has spread throughout the whole architecture. Then once again, you end up changing many different things to fix one bug.

Maybe they make it easier to understand the system? Well no. Generally, more layers means more explanation. Bringing a new person up to speed means explaining a lot more things, some of which will seem quite abstract compared to the problem to be solved. If you're lucky, that person can just be told about one layer - but in many teams each person will need to work on and understand all the layers. In this case, you've greatly increased the amount of training required.

So where is the benefit?

Lets look at some examples that work. TCP/IP is a famous example. By dividing the idea of a "network" into several layers (TCP, IP, and physical), each with a well-defined purpose, the designers made a practical, working software architecture that runs the world today.

Interestingly, the OSI group also defined a multi-layered network. In fact, they defined a lot more layers! A full set of seven layers. But infamously, this resulted in an unwieldy monstrosity that was used only in certain niches and eventually abandoned for the much more pragmatic TCP/IP.

So it seems that more layers is not always better. There must be some way to decide ...

TCP has a very clear and useful purpose: to make an unreliable network (IP) look like a reliable connection. Wow! Very useful (although also, amusingly leaky). IP has a clear and useful purpose: to route blocks of data over the network and find their destination.

So I would go back to a very simple principle from Agile methodology, which is this: YAGNI. You Ain't Gonna Need It. If you cannot, right now, see a real purpose of that layer - a purpose that has meaning in achieving the requirements of the system, not just an internal purpose - then you won't need it, so building it will be a waste of time.

The default settings for Visual Studio 2010 are far from optimal. In our particular moderate-sized solution, every build or run took 2 minutes or more - yes, even incremental builds.

Admittedly, VS2010 is rather bloated in it's design, some of which adds value (sophisticated intellisense) and some which doesn't (the editor being in WPF). The biggest issue for us was we are using SQLMetal to generate schema automatically from our SQL; this results in a lot of database work every build. Perhaps "don't do this" could be helpful...

Nevertheless, the following settings will improve that considerably. We got our incremental builds down to a few seconds. Many people have posted some of these settings, but perhaps this will collect some more useful ones together.

Visual Studio IDE settings

Go to Tools - Options and set the following:

1. Projects and Solutions - Build and Run.
   1. Check "Only build startup projects and dependencies on Run"

This will prevent it building all the projects all the time!

2. Environment – General
   1. Uncheck "Automatically adjust visual experience based on client performance"
   2. Uncheck "Enable rich client visual experience"
   3. Uncheck "Use hardware graphics acceleration if available"
3. IntelliTrace – General
   1. Uncheck "Enable IntelliTrace”

This disables a specific tracing technology most people don't use, that adds major overhead.

4. Environment - Startup
   1. Set "At Startup" to "Show empty environment"
   2. Disable "Download content every..."

PC setup

1. Get an SSD!
   We use an Intel SSD caching system, which is improves our build times by about 50%. Specifically it's motherboards with 20GB mATA SSD drives, using  Intel Smart Response Technology  (or is it called Rapid Storage Technology collectively?). We have it set to "maximized" mode, which means it caches writes as well as reads. I suspect, but haven't yet tried, that a pure SSD would improve it even more.
2. If you can't get an SSD, defragment drives properly.
   The built-in XP one might not good enough to defragment free space if the drive has become heavily fragmented at some point! I used: Auslogics Disk Defrag.
3. Exclude virus scan on network, work and VS folders.

Project specific

Unload projects if you're not maintaining them or are unlikely to affect them in your work.

Inserting Cross-references in Microsoft Word is still a painful process, even in Office 2007 (have not checked 2010). The main issue is that it makes you insert the various parts of a useful cross reference individually.

If you want a useful link you probably want it to be usable both in print, and on screen, and obvious as a link in both cases. Something like:

(see 4.5.3 Maintaining Your New Bicycle, p9)

To do this requires that you do three reference inserts, each time selecting the heading you want to link to, and the correct type of link, and also typing space, comma and p manually in between; and then select the whole lot and change the style to blue, underlined. This is very tedious, for something that you probably want to do a lot.

So here's a Word macro to insert nice cross references, all in one step. (Hosted on my website, probably not full of viruses)

Installing it in Word is pretty heinous. Let's say you want to install it into your Normal template:

• Unzip the InsertNiceXRefMacro.zip file to a folder
• Start Word
• Press Alt-F11 to access the "Visual Basic for Word" editor
• Right click on the Normal template, and click Import File
• Browse to where you unzipped the file
• Click on the NewMacros file, then Open.
• Right click, and Import File again, this time selecting SelectHeadingForm
• Click save and exit.

I personally link this to a keyboard command, Ctrl+Alt+X which comes to hand fairly easily. But you could also put it on a button.

With this Macro, I merrily insert useful references all over the place!

Issues:

• If you "Update Fields" to bring the links up to date, then the blue, underlined formatting gets lost. Any tips on how to solve this?
• If there are any blank heading entries in your document (where even the number has been deleted, but the style is still Heading X), then they will appear in the "select heading" dialog this macro uses, and will mess up any links after the blank. To resolve this, just remove the blank heading entries. Interestingly, Word's Table of Contents somehow resolves this, I don't know how.

Microsoft Visual Source Safe (VSS) is a version control program, but is not a very smart product.

For many years I have been confused by VSS' way of branching. Conceptually a branch is pretty simple: I want to take a copy of this folder at this point, and maintain it separately (typically for bug fixes in a release). Whilst keeping a record of the shared history up to this point.


But VSS doesn't make it simple. Instead of efficiently handling this use case, it pushes the effort and thinking back onto the user. VSS claims to offer "label promotion, pinning, sharing and branching". However, these four "features" really only result in one useful way to do branching in VSS.

• Sharing is simple looking at the same file in two different places in VSS. By itself, it is only useful for files you want to share in two parallel projects, and could equally well be achieved by having a single shared directory in your source code.
• Pinning is simply a way to override what comes out of a project/file if you use "Get Latest Version".
  
  • Pinning is used by VSS to implement what I will call "retarded branching". If you share a file from the  History dialog (but not via drag and drop!) then by default the newly shared files will be pinned and thus remain identical to the original branch point, even as changes continue on the trunk.
  • However when it comes time to make a change, instead of just checking out, editing and checking in, you now have to give VSS the "clue" that it should now branch each file. This causes VSS to do "real, but still retarded" branching.
• Actual VSS branching is very crude. It simply takes a copy of the entire file, along with it's history (but minus it's labels!). This technically meets the definition of an actual branch, but is extremely inefficient because the history is physically duplicated.

Between pinning and branching you could have "semi-retarded branching", where unchanged files remain as shared, pinned references to the original and changed files are actually branched, but only at the cost of you having to manually branch each file you want to change, and confusing everyone with "pinning".

All this is just a boatload of pain for an operation that should be simple, which is why everyone but Microsoft recommends simply branching everything at the start.  The simplest way to achieve this:
1) Make the new folder you want to put the branch copy in.
2) Right-click on the original folder, and then drag to the new branch folder
3) Select Share and Branch from the popup menu
4) Check "recursive" in the dialog, then OK.
You can also do this from the project History, but this involves a lot more steps.

This then works, but uses vast amounts of disk space to copy the entire archive.  So it is actually more efficient simply to make a new folder and check in everything again for the branch.

Since Microsoft's engineers are clearly loathe to undo their clever Pinning feature, and the Document writers are vague on what is the point of any of it, the dialogs don't default to branching at the start, and the docs wax lyrical about the multiple "options" available". You can safely ignore it all.

Microsoft themselves have basically pushed VSS into the background, and created Team Foundation Server instead. TFS is much better engineered, but a bit clunky for simple uses.

uTasker is an embedded TCP/IP stack. It has a lot of features and ports to different microcontrollers. It's free for hobbyists and cheap for commercial use.

However... you should be aware of what you're getting with uTasker. After a painful introduction to it's limitations, we have figured out several important points of uTasker:

• It has not been tested thoroughly - on any platform, let alone all of them. Testing is adhoc, not against a complete test suite. This is probably to be expected for only $500, but it's not clear from the outset.
• The result is that it is quite buggy for many cases where it has not been tested by some existing customer. If you're using anything out of the "ordinary" (see below), you are very likely to find major bugs. We found major critical bugs in several parts of TCP, such as breaking the connection or losing sections of data, and some serious limitations in DHCP.
• The support is responsive, and to his credit Mark has fixed the bugs we found.... but we had to find the cause of the bugs in detail, effectively debugging the code ourselves; Mark's contribution has been to fix the bugs once found. This is still a lot better than nothing, but diagnosing the bugs is often the hardest and slowest part.
• Mark has thus described his model as "open source with support". The stack you get is only as good as customers have tested it so far.
• Although responsive, Mark is not full time and thus cannot provide 8 hours per day of support if you need it for a critical bug. He will try his best ... but he's doing other things.
• Some of the fixes made caused other problems, because of the lack of a test suite for regression testing
• Documentation is scattered and incomplete, especially for the TCP API itself. At first we thought this was managable, but it wastes time to ask support instead of having basic information written down.

So what is uTasker good for?

I said above that you should be cautious if doing anything "out of the ordinary". What is considered "ordinary" for an embedded TCP/IP stack? Well, it seems that the usual uses for uTasker, and those that are best tested are those that are:

1. are on a local network, not over the internet
2. in a limited fixed hardware context, e.g connecting to a certain hub, not just allowing users to plug devices into any old piece of equipment
3. are strict client-server applications (e.g poll response, no full duplex)
4. have no major timing requirements (e.g. uTasker has a fixed 3 second TCP RTT)
5. don't require high reliability - uTasker is just not tested to such standards and won't achieve them.
6. use the supplied uTasker applications (web server, FTP). These applications limit themselves to the well-tested aspects of the TCP/IP stack.

For example, providing a web configuration on your device, or having the device send data to a server on a slow schedule. Basically, non-critical things which can fail or go slow, and be retried by a user or on a timed basis.

Our application did not match much of this, and thus we were seriously tripped up by bugs and limitations. If possible we will drop uTasker and use raw Ethernet, but at the most we will end up using UDP... which of course is not making much use of uTasker...

Final words: Embedded TCP/IP is still hard, if you want it to be fast and robust

To be fair, it seems we underestimated the cost and difficulty of a working embedded TCP/IP stack. It now looks like you have to pay a lot, be prepared to take a big hand in debugging it yourself, or limit yourself to simple stuff. It's possible that uTasker is the best you can get for the money... but it's also possible that lwIP would be a better cheap option.

What if you want to pay a lot to get an already-tested, robust, fast TCP/IP stack? Well, I don't know. I've seen good signs that Micrium uC/TCP-IP takes quality seriously including some signs that they have certified against a test suite. But we haven't tried it. Keil also have a "serious looking" stack and OS (well, it has a serious price!) but they are not very responsive.

So we went looking for an embedded TCP/IP stack ...
But it's not too obvious what to choose. There are a lot of options but many seem inappropriate for one reason or another; usually they are unsupported or incomplete.

Keil make a combined RTOS and TCP-Stack: RL-TCPnet. This looks fairly comprehensive (it ought to be, Keil are owned by ARM now!), but expensive and support doesn't look that great despite being an expensive commercial product. In a similar vein, NicheStack is expensive but looks good (there's a free version NicheLite, provided by NXP; but that appears to be practically unsupported). Well, not having the budget to blow US$8,000 on a TCP stack we moved on...

We looked at uIP, which seems ideal for embedded projects with it's simple API and lack of depencies. But, uIP is not being centrally maintained; the author has moved on to other things, and a number of branches have been created (e.g. FreeRTOS, Contiki). We investigated for a while but it became clear there are a several quite significant bugs, and that parts of it are not really complete, such as the DHCP implementation. So while uIP seems suitable, it isn't in a state that can be relied on, unless you are prepared to fix it yourself.

So we went looking further. lwIP (Lightweight IP) was written by the same guy as uIP, but the difference is that an active community has taken over maintenance and actively fixed and improved it. lwIP looks like a good option, although it is significantly larger than uIP. It is also one of the most full featured, including automatic private IP (APIP, or zeroconf) and a socket interface (which requires an RTOS). We concluded that lwIP is one of the best. For us, there was only one downside, that it didn't have a working implementation for our microcontroller.

We also found uTasker. This is a small company - in fact, it is one person part-time, The best part of uTasker is that Mark Butcher is very active and responsive in his support. It also has working ports to many common microcontrollers and IDEs. It's a commercial offering, but the price is reasonable at less than US$1,000, and you can use it free on non-commercial projects.

We picked uTasker, but came to partially regret it, spending a lot of time fixing bugs and finding limitations. More to follow...

NXP is the new name of what used to be the chip-making part of Phillips. They sell chips. Like the LPC2366 microcontroller, which we are trying to use. An essential part of selling a chip is supporting people who have technical questions about the chip.

But NXP don't seem to feel that's necessary. I have two issues: one low priority that has been open for two MONTHS without any response, and a new high priority issue (yes we really want to use your chip NOW) that has been open for a week without any response.

Given the way the electronics design process works, this is pretty much enough to put me off ever using NXP again. Maybe they support big customers better, but for small companies like us they are too risky if they can't be bothered answering questions.

Oh, and that's not mentioning that there appears to be a catastrophic failure in the Ethernet part of the LPC2366. It would be nice to be able to fix that. Like I said, we want to use their chip.

I could also grumble about the fact that they supply an unsupported TCP/IP stack . Or the fact that their search engine sucks, returning stupid results for LPC2366. Or that their website sucks, using clever dynamic HTML ("liquid mode" ) that doesn't work at all on simpler browsers without Javascript.

Since there doesn't seem to be any notice of this on the web, I thought I would post for the record.

My main bank, BankDirect, recently sent me a paper letter saying that they have had a security breach, that my Visa credit card details may have been compromised, and that they wanted to send me a new credit card. I suppose it is quite likely that the breach occurred at Visa or a credit card processor or merchant. But they didn't give any of that information so they can have all the blame for now :)

I'm not that surprised as many banks throughout the world have had such problems with credit cards, due to sloppy security practices. Still it's a bit discouraging when it happens to you. 

To cap it off nicely, BankDirect woke me up one morning to setup a new card, and then spent almost half an hour trying to figure out their own systems to complete the process :(

The operator  made some interesting comments about how they get mostly ASB calls and BankDirect is not very common (BankDirect is basically a side venture for ASB). I've been using BankDirect for about 10 years and never found anyone else who uses them, although quite a few people know the name from the ads earlier. Cashiers are often a bit intrigued by the unusual card. I suspect that BankDirect is only a minor interest for ASB now, because most banks in practice operate largely over the internet and phone anyway, so it's nothing special. Recently they dropped the interest rate on their "high interest" transaction account to such a low figure it's useless (from 6% to 1.5%) which made me wonder if they are trying to kill it off. Well, I took their hint and am now transitioning to TSB. So long, and thanks for all the fish BankDirect!

Recently I've taken to using Google Chrome as a browser on my netbook, and I gotta say, it's a MUCH better fit than Firefox.

First of all it seems considerably faster - especially to startup. Firefox has always been a hog, loading it's custom javascript/XML GUI, and relies heavily on a fast hard drive - which just isn't there on a netbook. Chrome loads very quickly.
Second, it makes considerably better use of screen space, without losing features. I used to think a menu bar was important as a standard GUI item, but it wastes a lot of vertical screen space. So Chrome's minimal menus on the toolbar is better.

There are a lot of nice touches in this area. For example, the Firefox status bar shows useful information like the URL that a link points to (when moused over) but you don't need a whole screen width permanently devoted to it. So Chrome just pops up a minimum width bar to show URLs when needed.  

The multiprocess architecture of Chrome sounds good technically, but I haven't rely found anywhere that it makes a difference, perhaps because Firefox is quite reliable anyway. In theory it would make it more responsive, but the bottlenecks on my netbook seem to be in processor time and hard drive, so a busy tab still causes pausing in other tabs. It does cause Chrome to use quite large amounts of memory, but that doesn't seem to be a problem in practice. I've had a few glitches using Chrome anyway, so in practice it doesn't seem to help reliability.

Another nice touch is that new tabs appear to the right of the current tab, not on the end, and there is a new tab button in an obvious place.

Finally, a pet peeve of mine is that Firefox splits the URL and search fields. This just seems pointless. Even with addons, it seems difficult to get a single integrated search+URL bar that works as well as Chrome's.

There are a few things that seem clumsy, like having to hold down on the back button to see further back; the small separate button on FF seems like a better solution. Also the space wasted at the top of the screen above tabs seems pointless. Site compatibility is not quite perfect either, which is unfamiliar these days (used to be very common in early Firefox days).

The one thing I seriously miss is the Flashblock plugin. I think there are some solutions for Chrome now but they don't seem very official. One Flash ad on a page usually turns things to mush on a netbook. Flash is just such a hog the only solution is to block it by default.

In general, Chrome developers seem to have thought harder about the details of the UI than Firefox guys, who still seem hamstrung by their immense technical architecture. Kudos Google! 

I'm currently using iTunes 9 for XP, on an MSI Wind NetBook.
While it "works" iTunes is very sluggish. Often there are long pauses while it does something.
I think iTunes is pretty good and mostly like the store.  I don't mind it having lots of features in it because Apple unlike many companies manages to keep it simple to use.

But I really, REALLY wish Apple would work on performance.

PS. The Store user interface is unfortunately, also too wide to fit on the Wind's screen (1024 x 600) without scrolling. A bit painful.