Evaluating the health of a software development unit

[ Out of scope (but extremely important): Are we producing the right thing? ]

If I were to evaluate a software development unit, I would ask the following questions:

(measures of current health)

1. How long does it take after an new idea is conceived until it starts being used by end users?
2. How long does it take until a change request or a bug is discovered until it is deployed?
3. How consistent are these times?
4. How many errors are being deployed to production, and how are they discovered?

(measures of continuous/improved health)

1. Are developers growing?
2. Are processes being improved?
3. Is technical debt managed?

 

To probe further as to where the areas of improvement lie, I would investigate the following key attributes of the product:

1. Maintainability (e.g. coding standards, test automation, documentation)
2. Release management (e.g. CI/CD, traceability, planning, configuration mgt)
3. Risk management (e.g. single-point-of-failures, technical debt, regulatory compliance)
4. Quality management (e.g. testing, code reviews, repeatable procedures)
5. Serviceability (e.g. logs from production (errors, statistics), handbook, su configuration)
6. (Security (e.g. scans, audit, access mgt, policy))

Making (new year's) resolutions - a framework/categories

How should one go about setting (New Year's) resolutions? I tried last year to set myself some goals, it was useful. They were not overly ambitions but gave me some push to complete certain tasks which gave me a sense of accomplishment at the end last year.

So this year I decided to give it another try, but this time I went about it more methodologically (big word!). After several iterations I ended with the following categories. I am pretty satisfied with the categories, think they create the right framework for settings goals with the focus on the long term.

Two categories go to creating a well functioning 'machine' necessary for everything else:

• Physical well being/performance (ísl. líkamsrækt).
• Mental agility (ísl. hugrækt).

 One category goes to purpose or basic need:

• Connections (family and friends in particular, those that will last to old age).

 One category goes to security:

• Professionalism

The last category goes for guiding the decision making (for this journey to old age)

• Values.

Setting up the role of testing

We have tried assigning testing to three different roles: The developers themselves, a tester test-in-the-loop, a quality assurance officer.

Presupposing that manual testing is a good thing, we have discovered pros and cons for the different setups.

Developers as testers

Having the developers cross-test each others' issues has the benefit of transferring knowledge of the code, of best practices and of improving the art of communication [arguably this same objective can be fully covered with peer-reviews --Ed.].

Developers however are not equally good at testing and probably not as good as a trained tester. It might not be the optimum use of resources. Some even find testing boring and that it breaks up their coding-flow.

Tester in-the-loop

Taking the task of testing from the developers and placing it on a dedicated tester has the benefit of localizing the responsibility of the testing. A tester puts up his/hers own testing environment, and improves his/her processes through greater exposure to testing. The tester has a slightly different approach to the issues, seeking out those special corner cases that lie off the happy-path.

There is a danger of that the developers might grow sloppy, assume that the tester will catch all their errors. And if the throughput of developers and testers are unbalanced, bottlenecks can develop.

Quality assurance

Zooming out, what testing needs to provide is a bug-free customer product. So ultimately what is most important that is tested is the delivered functionality of the product. This reflects back on how requirements are specified, in particular that they are testable. Keeping issue testing with the developers and higher level functional testing with the tester takes the tester out of the loop and places him/her on a parallel track with the developers.

Errors however typically surface later and in Scrum often during another sprint. This of course can be costly and makes tracking issues and burndowns a bit more tricky.


The verdict was not in when I left my last employment, but my feeling was that testing should not be a reactive role (in-the-loop only) but more of a proactive role, and that the tester should be able to proxy the product owner to some extent.

Managing and leading software product development

I have been contemplating the roles of the development manager and the product manager in software development. In particular who has what responsibility?

I recently read something that resonated with me, that leaders are needed to bring about changes and managers are needed to create order out of chaos (complexity). The point being made was that both roles are legit, but require different skills and have different end-goals. The tools of the manager are processes and structures while the leader uses motivation through values and emotions. The goal of the manager is to accomplish the plan and that of the leader to achieve the vision.

In software development it is necessary to decide on doing the right thing and then do it right. To achieve this we both need leadership and management. But how is that best put in place? It is tempting to consider this best done by having a leader-manager. I, however, think it is generally too much to ask for in one person (even though they no doubt exist). In my view the role of the development manager requires more management than leadership, and that the opposite is true for product management.

Development managers should not be pure bureaucrats without vision and the ability to motivate people. They might have the goal of bringing structure to the work but they will not succeed unless what they propose is accepted by the organization and the teams in particular. To accomplish this they might need to find novel solutions and be strong in communicating and negotiate for those. The development manager must not buckle the leadership of the product manager by wielding his official authority (embedded in the org-chart).

What I am getting at is that I don't think that management serves the derogatory conjugation it has had. Leadership and management are both necessary ingredients to successful software product development.

Team structures

The last 11 years we have tried out different setups for the development teams, looking for the one right setup. Actually, now, I think the right approach here (borrowed from others) is to address team structures as a product development. In particular, to do incremental adjustments ('developments', 'experiments') a keep track of these through release notes and retrospects.

This is an attempt to (posthumously) create these release notes/version history.

Version 1

We were three developers and we split the responsibility based on architecture. There was no over-all lead, but we had an IEEE requirements document that we with others had created.

This was a time where responsibility was very clear and code-ownership strong.

Transition driver: It did not make sense to drive all projects with a single developer owner, it would be too slow. New functionality needed to be implemented by a group.

Version 2

We instigated Scrum, we had the product owner and scrum-master roles. A single team.

I think everyone felt good about having clear task lists. Also clearing responsibility for non-code work, such as doing meeting notes, setting up and maintaining the development stack, taking budging decisions.

Transition driver: We started a new code-base, a new issue tracking project, and dedicated split of the developers. But kept the single Scrum team structure. The thought was that it would create too much over-head having two teams, also we thought the knowledge transfer would be best served for both groups to continue working as a single unit.

Version 3

It became obvious that during planning, demo and retrospects half the group kept silent while the other group went through their development and visa versa. Time was being wasted, people were bored. So we split into two Scrum teams although keeping a single product owner and scrum-master.

Transition driver: We started getting more and more load from the service department and we were starting seeing more customer specific development projects. We started out by allocating fixed amount of time and a dedicated developer for 3rd level support issues. We called this role "batman" at first and later "the hat", it rotated between the developers, one having the role for the full sprint.
It was rather artificial allocating a fixed time slot for 3rd level support since it was rather unpredictable how much time would be needed. Frequently this would mess up the burn-down chart and could deteriorate the commitment of completing sprints.

Version 4

We created a specialized 3rd level support + custom development team, called Quicksilver. We pulled out all bug and custom development issues from the Scrum boards and put it on Quicksilver's Kanban board. We populated the team with the two most senior developers. They also got the go-ahead to do the refactoring they thought necessary to ease the future maintenance burden.

Transition driver: The developers on the Quicksilver team resigned. 

Version 5

We restructured the teams into three. This was actually the second attempt where we went from one product owner to three. We discussed quite a bit if we should do this split based on code-bases/architecture or if we should try to do it based on cross-code-base features. The first option won because we thought code ownership was more important then having more flexibility in creating teams around end-to-end functionality. We were still doing quite allot of maintenance and that usually meant going into closely contained areas of the code. However, the latter option has a greater appeal to top-management because they figure that it will give them greater throughput of new (sell-able) features.

There are some versions missing, one is where we when for the first time from a single product owner to three. And first placing them within the teams and later moving them into the sales department putting greater responsibility on the teams. The distribution of responsibility between product owners and the teams is a topic belonging to a blog of its own.

Also missing are our attempts in incorporating testing into the development, by letting the developers do it, to having a tester picking up issues as they complete within the sprints, to have complete sprints tested. This is also a topic that requires a dedicated blog of its own.

Lastly, there is also how the team organizes its work internally. Should everyone be able to do everything, or does it make sense to have some specialization, if, then to what degree? We have done some experimentation here as well. Blog++.

Useful measures for the development manager

After establishing a functional structure for the development department this structure needs to be monitored (re-evaluated) to see if it still fits its purpose. For this monitoring task some measure are needed. Being an engineer I tend to gravitate towards quantitative measures rather than qualitative.

At first my focus was internal to each team, in a typical Scrum-master style, basically tracking the burn-downs within the sprints. We used hours to do the estimation, so the focus was very tight on hours delivered, as opposed to functionality delivered.

This way of estimating has been helpful in improving the planning of individual issues before heading into programming, it creates a communication platform for the team. It also gives the product owner some estimate of ETA for the increments to the product. However, it has not functioned well as a motivator, it has always felt rather artificial. Having an actual deadline where the team commits to someone external has been a much bigger motivator. The teams have been moving from these time-based metrics to use story points, the verdict is not in yet on this new approach.

Recently my focus has moved from internal team-metrics to a broader departmental scope. The metrics I track today are also based on hours, but accumulated per projects, teams and individuals. The goal is to track:

i) Time spent vs budgeted (estimated) per new development. Budgets are approved and if over-run need to get re-approved.

ii) Time spent per activity: product maintenance vs new product development vs custom development vs. dev-ops. These statistic can tell a lot regarding that state of the code-base (too much maintenance?). If new development is being starved. Costs of dev-ops should not be confused with the cost of new development and maintenance. Same thing goes for custom development that should generate the right profit margin.

iii) Time spent per team and per individuals is important to know in order to determine the load on employees and if it is being fairly distributed.

These metrics are calculated monthly.

Becoming a development manager

I wanted to document how I moved into management, for my own benefit and hopefully others'.

Becoming a development manager

In a growing company the development manager typically moves from being a programmer into management. My transformation went hand in hand in having the lead in creating a structure in the development department. In order of appearance:

1. programming (adding to existing code/architecture)
2. architect-ing (re-tinking the architecture)
3. product packaging (Wix-ing, release notes, guides)
4. setting up rules/guidelines regarding code and documentation (developer's guide)
5. selecting and setting up the development environment: version control (SVN), build systems (CC), documentation systems (Confluence), issue management (Jira), test systems (NUnit).
6. formalizing and detailing a development process (Scruminze, even ISO 9001)
7. hiring (setting up the hiring and on-boarding processes)
8. vacation planning, annual employee interviews
9. budgeting

.... these tasks overlap in time and gradually go into 'maintenance mode'.

Being a development manager

After setting up the structure the next challenge is to maintain a functional system, this means:

• evaluating the current split into teams
• evaluating if there are skills missing
• evaluating if work should be done internally or externally
• evaluating the processes
• evaluating employee happiness and motivation

The next stage in development for the development manager is moving from ad hoc fixing the existing structure to create a meta-process for this. The driving and constraining forces for adjusting/transforming the structures are (in no particular order):

• internal organizational changes
• changes in best practices
• changes in technology
• availability of skills
• employee motivation

What a development manager is not

I belief that job descriptions should be as complete as possible and this sometimes means that they need to state what is not included. Of course this responsibility then need to lie elsewhere or the organizational chart is incomplete.

For me the development manager has the main responsibility of doing development right. It is not doing the right development. These are two huge tasks in their own rights. One lies with the development manager the other with the product manager.

Force those windows to become visible

HOO-ah! I have finally found the utility I have so long sought!! I have had the problem of loosing windows after un-docking my notebook which is connected to a secondary monitor at work. Some programs are real smart and reposition themselves to the primary monitor, other programs can be persuaded to the primary monitor with the "ALT-SPACEBAR M" trick. Then we have the stubborn ones that refuse to budge. Now finally I have found the tool to force these programs onto the primary monitor: ForceWindowVisible !

How come that Google only gives 4 hits when asked to lookup "ForceWindowVisible"?!?

Debugging stored procedures in SQL Server 2005

I have been using the SQL Server Business Intelligence Development Studio to debug some stored procedures that I work with. It seems impossible to view the content of the temporary tables during debugging sessions. This fact limits the tool's usability severely for me :(

Why is it so difficult the get a confirmation of this limitation searching the internet??

Now I made the temporary tables permanent (removed the #) and manually drop them after each debugging session.

Using branching in Subversion

For small changes a developer will typically work directly on the trunk, but for bigger changes the developer will typically work on a private branch. This is best done in the following way:

• Create a branch (at the trunk-level) and name it something like aris-private-branch.
• Select the part of the trunk's sub-tree that will be affected by the branch and svn switch that sub-tree to the new branch
• Make changes locally and test them
• Commit the changes to the branch
• When all is done, merge the changes to the trunk:
  • svn switch back to the trunk's sub-tree
  • svn merge using the initial revision of the branch and the head revision of the branch
  • Commit the changes (it is prudent to check that you are not eliminated other people's changes that were done to the trunk while you were working on your private branch)
  • Delete the private branch (through the repo-browser)

For changes spanning a long time, it might be necessary to merge from the trunk to the branch to incorporate other developers' changes. In this case it is simplest to keep track of the revision number of the merge from the trunk so that next time only changes in the trunk from the last merge are merged to the branch.

WiX

What a horrible, yet fascinating, thing WiX is.

Executable context

Sometimes it is known that certain files (the .exe.config in particular) that are necessary for the running of an application reside in the same folder as the executable. Now, if the executable is not run from the directory where it reside it can run into problems locating these necessary files. In this case the following code comes to the rescue:

static void Main()
{
string directory = Path.GetDirectoryName(Application.ExecutablePath);
Process.GetCurrentProcess().StartInfo.WorkingDirectory = directory;
Application.Run(new Form1());
}


ILMerge

I just found out about this utility/possibility. Thought it was so interesting that I should make a note of it here: ILMerge. It can be used to take an executable and a dll it depends on and package them into a single executable.

ILMerge.exe /out:complete_app.exe app.exe app_lib.dll

I have one scenario where this is useful for me: I have a program that has both a console and gui (windows forms) front-end. In order not to duplicate code I have 3 project, one for the console part, one for the gui part and one for the common code. In a Wix installer script I need to reference both the gui project and the common dll, but by inserting the dll into the gui executable I only need to reference a single executable in the Wix script :)

True, a marginal improvement, but, still, many brooks make a river :)

Utilities

I thought that in case my computer crashed I should have listed up all the useful utilities I have gathered so far. I have tried this before, but this list is more complete.

• Sysinternals suite (filemon, regmon, procexp, tdimon, tcpview, handle)
• SnippetCompiler (to do short-lived experiments)
  
• baretail (to monitor logs)
• CCleaner (to tidy-up my mess)
• ipscan (to get to know my peers)
• Reflector (to see how others do it)
• Google desktop extreme (find that lost info)
• Agent Ransack (what is the Windows search useful for?)
• ScreenHunter (screen snap-shots)
  
• HHD Free Hex Editor
• EditpPadLite (there are Notepad2, Notepad++, Crimson, etc. as well)
• IrfanView (renders all image types)
• Recuva (undeletes that important file)
• Ad-Aware (to clean up my mess as well)
• Toad for SQL Server (actually went back to SQL Management Studio, old habit I think)
• SQL Manager 2005 for SQL Server Lite (intellisense when creating queries)
• SharpDevelop (interesting, but since I have VS I think I stick with that)
• 7-zip (actually usually use the build in Compress in XP)
• FileZilla (when I need a ftp server in an instant)
• Gimp (to edit my Display picture)
• Picassa
• Firefox (to be complete)
• SmartSniff (to pry into my neighbors business)
• TreeSize (find that disk-hog)
• Launchy (faster and more intelligent then Ctrl+r)
• WinMerge (compares files and folders!)
• Reinlendar (nice desktop calender, but surprisingly hungry for memory)
• ForceWindowVisible (a must-have for people joggling between using 2 and 1 monitor)
  

I am no doubt forgetting some. Actually I am very often install some utilities that I think sound great and the forget about them :/

logging/tracing guidelines

I use log4net for logging/tracing, this is the current guideline:

• Debug – Used for the developer him-/herself during development
• Info – Used to leave an execution trail of the code. Useful for orientation after an error has occurred run time, in particular where the stack-trace does not capture the complete history of the operation
• Warning – Used to flag a problem that is external to the code in question, e.g. related to incorrect calling parameters. It is not perceived that the next call will generate an error.
• Error – Used to flag an internal error, when a unperceived problem occurs within the code, the operation fails
• Fatal Error – Used to flag an error that makes it impossible for the component to continue operation. This typically happens at startup when the component is not able to initialize correctly, e.g. because there is no connection to the database which contains the component’s configuration
  

Documentation

I was going through the design documentation of our project: It started out 2+ years ago with a requirements document (following the IEEE-830-1998 standard) from which we created an architectural design document (following the IEEE-1016-1998 standard) which we broke up into detailed design documents, one for each component of the system. I think we can truthfully claim that the first version of our product was fully compliant with the high-level documents. But in the following bug and minor releases not all high-level documents got updated, and now we are at our third minor release and the documents have become seriously out of synch. Maintaining the original documents is a headache and now the only time you will find me reading them is when I am checking on how outdated they have become, never to refresh myself on some high-level detail. Hmmmm.... So I started to rethink (as no doubt many before me) what the purpose with these high-level documents is, and the remainder of this entry is meant to ponder that question.

Requirements
With regards to the requirements documentation, I think that at the start of a project it is helpful to have a requirements document, the IEEE standard one is good to follow so that that no details get left out. It is useful to have one complete document to keep an overview of the functionality of the whole system. It is also a good medium for circulating the requirements within a reviewers group. However, when the development is under way, I think that the requirements document should be phased out (not maintained) with an issue management tool. The old requirements do not necessarily need to be transferred to the issue management tool, but all new requirements should be entered into it. Using an issue management tool makes project and release managements much easier: each issue gets a priority and a designated developer, and a log of its history (changes and comments) is maintained. And when it is time to do a new release it is easy to determine the current status of the software, what has changed since the last release and what has been left out as open issues.

Code Design
I think that it is necessary to keep the code as readable (self-documenting) as possible. It is necessary to include documentation for the following items. This might be best done within a #region of the code or perhaps in a separate document that is placed close to the code, convenience dictating the choice:

• Scope: What this component is to do and what not (I am assuming a component based design)
  
• Design: What high-level structure (pattern) was selected to solve the business problem
• Design alternatives: What other structures were considered
• Design rational: Why the design alternatives were rejected and the selected design chosen

The reason why these items should be documented is to facilitate the next developer to know the thinking of the original author (be it himself re-acquainting himself with the code some years later or a completely new developer).

What should not be documented:

• Class diagrams, since they can be automatically generated, if needed (using Visio e.g.)
• Database diagrams, since they can be automatically generated, if needed (using Management Studio e.g.)

In general, the documentation should be kept as sparse as possible, in the spirit of YAGNI and KISS.

Running VisualStudio in the command line

It gave me pleasure to learn how to run the VisualStudio in the command line:

c:\devenv solutionfile.sln /build debug /project subproject

This can be helpful especially if I just want to build one project in a very big solution where it would take forever for VisualStudio to start up (especially with Resharper installed).

Running an assembly from an intranet share

I needed this to be able to access an executable on a network share without opening up for executing all Intranet assemblies. The recommended way is to use the strong name for the assembly, this was surprisingly easy (this info can be had all over the internet):

1. Create a key: sn -k mykey.snk
2. Add "[assembly: AssemblyKeyFile("mykey.snk")]" to the AssemblyInfo.cs
3. Compile :)

To allow this particular assembly to be loaded and run:

1. Run ".NET Framework 2 Configuration" (to be found under Administrative Tools)
2. Navigate to My Computer->Runtime Security Policy->Machine->Code Groups
3. Right click on All_Code and select New and give it some name. Next
   
4. Select "Strong Name" and click on Import
5. Select the assembly/executable in question. Next
   
6. Select FullTrust

OO vs Unit-testing

One reason for why unit testing has bothered me is that it goes against an OO principle I was thought at school, namely that every aspect of the code should be kept as private as possible.

Roy Osherove has a recent entry on this subject. Roy is an enthusiastic unit-tester, I am still in doubt :)

UPDATED 24.7.2007
Bruce Eckel's thoughts on OOP.

TDD presupposition

I am pondering the validity of the following statement:

"The theory is that testable code is better designed ... If the class is easier to test, it is a better design. The test first paradigm just forces you to use good design. It makes a good design the path of least resistance." -David Hogue

I know that this is taken as a given in unit-test camps, but is it?

UPDATED: 1.1.2008
I found out that I am not alone in not being convinced about designing for testability. The creator of TypeMock feels that designing for testability does not comply with YAGNI.