Pair Design

A lot of times, when writing software, a pair programming approach is used. It's an agile software development technique in which two programmers work together at one workstation. At any time, one is the "driver" and the other "navigator", and the roles are switched at intervals. Basically the "driver" is writing the code, and the "navigator" is simultaneously reviewing the code in progress.

I got introduced to this approach during my undergraduate, and we had to use this approach whenever we were working on an assignment in groups of two. Pivotal labs seems to be using pair programming approach quite effectively. I am not sure which other companies use this approach, but it seems the pair programming idea can be extended to the design process as well.

Generally an engineer (let's call this engineer the primary engineer) gets a task, maybe fixing a bug, improving a feature, or building a new API, all of which involves thinking and designing a solution. The primary engineer designs the solution, and the paired engineer is there to discuss the solution, address any limitations, good parts, and improvements before coding commences. The paired engineer will be a sounding board for the primary engineer to think aloud about the proposed solution. Of course, the roles will be reciprocating.

I am not saying don't evaluate designs as a team, this is very valuable, and is usually reserved for something significant. But for day to day design of solutions, having an assigned sounding board can help the primary engineer to think aloud about the problem. Each paired design partners will rotate, maybe every sprint, for example. This way, each team member will also get a chance to work with others, and an informal review process for designs will naturally set in.

Just a thought.

Psychology of Baby Steps Principle

I have borrowed the term "Baby Steps Principle" (BSP) from a colleague, a senior software engineer, at work. Full credit for the terminology goes to him, but the general perspective presented in the blog entry is my way of trying to organize some thoughts on it. Apologies for the length of this ramble.

We all, at some point or another, apply the "Baby Steps Principle" (BSP). BSP is the concept of developing or building things in an incremental fashion -- work on simple, constrained versions of a problem, and use the insights obtained to build up a solution to the complex, harder problem. This approach to solving problems is ingrained in our minds generally from school, irrespective of whether we come from an engineering or science discipline.

Even though we are quite intimate with this idea, often times we forget to use it. The following situation might seem quite familiar: you are trying to solve a problem, somewhat overwhelmed trying to aggressively apply many ideas; but your subconscious is telling you maybe BSP will work, which you completely ignore wasting hours, but ultimately resort to BSP and find that insight that helps land the perfect solution. A look at BSP psyche might help us get a better understanding on how to use BSP to our advantage.

The dogma of BSP, or incremental development, centers around this cardinal concept: get simpler versions working first, and build on that to get to the complex version. The simpler version of the problem is essentially a highly constrained version of the original problem. Once a solution to the simpler version is found, it can provide an insight into how to relax the constraints and approach a solution to the complex problem. Iterating on simplified versions of a problem is the key to solving the original problem.

We apply BSP in software development very frequently. For example, if you are developing a web MVC (Model View Controller) framework, why not build a bare or minimum version first, get it working, and then start building in your features to transform it into the end product. In scaling a system to 1 million users, we usually build a version for 100 or 1000 users first, and try to grow that number based on insights from building that relatively small scale version.

The above examples work because of a multitude of factors. But from among these factors, the key might be the motivation one obtains from small successful milestones. Successful increments give a sense of achievement, which motivates one to move on to the next step. The sense of achievement removes frustration of not getting anywhere as one struggles with the original problem.

In software, this greatly helps in debugging as well. It becomes less difficult to understand where the bug might lie after having reached a successful milestone. From the psychological perspective of the engineer, this translates to less stress since the process to identify bugs becomes easier, which in turn leads to better workplace motivation and increased productivity.

The examples are not just limited to software. If you have heard of dual decomposition, then you will see BSP also playing a role here. The basic tenet of dual decomposition is to decompose a complex, intractable problem into smaller tractable ones. The idea is to solve these smaller problems efficiently, and approximate the solution to the complex, intractable problem by building up from the solutions to these smaller problems. Figuring out how to decompose the original problem is itself a great milestone, being able to solve each of the decomposed problems is another milestone, followed by combining them cleverly to approximate the solution to the original problem and iterating.

From the cursory advantages outlined above, being able to apply BSP should be quite useful. How can we ensure we are using BSP when needed? The answer is quite simple -- listen to that subconscious or gut feeling. If you are feeling uncomfortable or overwhelmed while approaching a particular problem, it might be a good time to take a breather. Take a step back and think about constraining and simplifying the problem. Start working on the simplified version of the problem. Try to get some insight and incrementally remove constraints to reflect more of the original problem, and you are likely going to hit on an elegant solution to the original problem. Take advantage of the smaller milestones, and enjoy these successes as you work your way towards solving the larger, complex problem.

The concept of minimal viable product (MVP) from the Lean Principle also reflects the ideas inherent in BSP. Build up to a full, successful product by working on smaller milestones, evaluating them, and using the insights obtained to incrementally work on the next steps. I think we can all benefit from BSP, and it is just a matter of listening to that subconscious or gut feeling -- trust me, I am an engineer!

Principle of least coding

I would not call myself a software engineering veteran. As you read the piece below, you can think of it as a hodgepodge of ideas that have been in my mind for a while, and an attempt to obtain some clarity in the process of scribbling this. Please feel free to leave your thoughts and comments.

When I say "Principle of Least Coding", I want to be upfront and mention right away that I do not mean "obscure" coding. The term "least" should not be confused with obscure or obfuscated code. In a lot of cases, one can write up some exotic one liner that performs exactly the same logic, but the obfuscation causes more pain for future developers. Least coding is not about just a principle, but about elegance. If you are a coding veteran, you might very likely already know what direction I might be heading. A key point in my writing this post is to try to clarify and organize my thoughts on a collection of principles, which I refer to as Principle of least coding, which every engineer knows or have heard about, but still often times forget to incorporate in his or her daily endeavor of creation.

Least coding is the amount of code that just solves the problem of interest, in a comprehensive manner. There are no extraneous lines of logic that performs anything more than necessary to accurately solve the problem at hand. The code is as general as possible. Such code is also simple, which also makes it elegant. When you see such code, you will get a feeling of satisfaction and happiness -- you will feel the code just does enough to solve the problem at hand, nothing more or nothing less.

There is also an intimate relation between least coding and good design. When one designs a solution well, the amount of code that one has to write becomes minimized. Introducing configurability and minimizing coupling among lines of code or logic is an example of good design that also adheres to principle of least coding.

One can take a piece of code and actually convert it into least coding. This is what we usually refer to as refactoring. Refactoring slims down a piece of code or logic by subtracting out extra, unnecessary parts to make it lean.

Writing too much code that performs a lot of logic makes it susceptible to bugs. The best piece of code without bugs is the one that has not been written yet. As soon as the code is brought into existence in your editor, the chances of bugs creeping in become larger than zero. The point being, if you do not write too much code, then the chances of bugs creeping in that will come back and haunt you in the future are minimized.

So how can we go about writing code that is least coding? In general, when one works out the design (as has been apparent many times from personal and anecdotal experiences) well in advance before writing code -- which is something one should be doing instead of just sitting down in front of a terminal and start typing -- one can work a simple, generic logic which will minimize the amount of code one needs to write. Applying refactoring to the first draft of code can make it lean, transforming the final product into an elegant solution that adheres to the principle of least coding. An iterative thinking process, and a continual application of basic principles can make any junk code beautiful.

A retrospection on transition to industry

I have been working in the software industry for about a month now. So I thought this might be a good time to put down some of my thoughts in writing. To summarize, the transition to industry has been a very good learning experience so far.

Honestly, I am a complete newbie in industry. I did not even have an internship experience in industry before I took off from the Phd program. Although, I did have experience working on a large scale software project that helps scientists/researchers to perform large scale data analysis as part of an academic research experience, but working on software that gets deployed to production and, hopefully, reaps some profits in is a very different world. My experiences have been broadly in academia and in research in the past, and though my proficiency in various programming languages are more than adequate, after starting work in industry as a colleague with industry experts, I realized that there are many things for me yet to learn.

I have picked up a lot of things in a month, especially the myriad set of tools that are used to streamline the whole development to production process. I definitely have a much broader perspective on software now, and coupled with my background in research and academia, I feel more complete as a professional. My startup stint might have been the missing piece to my goal of becoming an accomplished computer scientist.

Computer Scientists are evolving?

I had an interesting conversation with a friend of mine today. As both fellow computer scientists (if I may call us so), our conversations shared the general 'geekiness' inherent in a conversation among this broad class of people. One thing we realized as we were reminiscing the good old undergraduate days and trying to figure out what our other computer scientist friends are doing, is that we all think not just about `elegant coding', but are also leaning towards the concept of engineering a useful product.

It appears, at least that was our view, many computer scientists nowadays do not just go and write software for a business, but bring in new product ideas and perspective in shaping the business concept. This possibly applies mostly for technology oriented companies, but computer scientists have started to fit into the finance and advertisement industry well. The point of this rambling is that computer scientists are not just code junkies: yes, we do write software, but we are also adept in potentially translating ideas that will provide a new perspective to new areas of venture, and to develop, maybe not the next big product, but the next 'very useful' product. Computer scientists are evolving!

How trustworthy are medical device softwares?

Even though I am mostly a Machine Learning/Computer Vision/Artificial Intelligence person, I do enjoy talks in other areas of computer science or mathematics. I think it is important to have breadth in ones field, which is somewhat contradictory to the dogma of being a PhD student -- obtaining ultimate depth of knowledge and expertise in a small portion of your field of interest. Nevertheless, I pretty much go to all the "Distinguished Lecture Series" talks with a quest for understanding the cutting edge work that is going on outside my area of "expertise" (and of course, the free cookies and coffee are a good motivation as well).

Enough rambling. I got to attend a great talk by Kevin Fu on the security and privacy of medical devices (such as pacemakers) and RFID based embedded devices. He raised some very interesting points about the current state of the art in the security and privacy of such pervasive computing devices, and possible research directions to address these points. His approach appears to be based on thinking of new ways of designing the software and protocols that are effective and energy efficient. Kevin is also noted in TR 35.

Dual decomposition: a recurrent theme?

Some thoughts on the idea of decomposition; any comments or discussions are welcome.

The concept of dual decomposition for inference in graphical models, particularly MRFs, is based on an insight that resounds in almost all aspects of problem solving in computer science. The insight is the ability to decompose a very hard problem into smaller, independent problems that are solvable, and then combine these solutions in a clever way to get a solution to the original problem. The elegance of this approach is that it can be used to solve inference problems with discrete variables that result in a difficult combinatorial optimization problem.

If we think back to one of our favorite sorting algorithms (at least my favorite), mergesort, the idea of "decomposition" pervades this elegant sorting algorithm. Instead of trying to sort a large array of size $N$, break the problem up into smaller sorting problems (the divide approach), solve each of these subproblems by recursively breaking into smaller and smaller subproblems, and then combining cleverly all these solutions (the conquer approach) to the small problems to get a solution to the original sorting problem. Of course, the dual decomposition idea for inference in graphical models has its own details, but essentially one can think of it as a form of "divide and conquer".

There are many great references that I can suggest if you are interested in dual decomposition ideas for inference in graphical models. It is quite amazing that the idea of "decomposition" has not only given us efficient sorting, but has lent itself in developing very strong inference mechanisms to address difficult combinatorial optimization problems.

Epilogue: I am including a few references that give a broad overview of graphical models and the dual decomposition framework. The article by Wainwright et al. titled "Graphical Models, Exponential Families, and Variational Inference" is worth looking at. I would recommend reading the first few chapters of David Sontag's PhD thesis. Sontag, Globerson, and Jaakkola have a book chapter titled "Introduction to Dual Decomposition for Inference" that might also be of interest. Komodakis et al. have a recent PAMI article titled "MRF Energy Minimization and Beyond via Dual Decomposition" which is also a good place to look. Another recent work worth looking is by Yarkony, Ihler and Fowlkes titled "Planar Cycle Covering Graphs". All these papers have a fair bit of mathematics.

CVPR 2010

I am in San Francisco right now for CVPR 2010. I was planning on writing up something on some of the interesting papers that I have seen here, but the conference has been keeping me extremely busy. Maybe I will write something up once I leave for home tomorrow afternoon. To summarize my experiences so far, CVPR started out not that exciting but in the last two to three days I have seen some excellent work. I am looking forward to this afternoon session on sparse coding.

"Write Blog"

I have to apologize for not being regular. It has been really long since the last time I wrote; things have been pretty busy, with research projects, full load of classes, grading, and all the usuals of being a graduate student. I have been able to attend a number of great machine learning talks as well, which I was planning to write something about, but just never got around to doing that.

I think I really need to put 'write blog' in my calendar.

AI taking over?

Here's something that is worth reading.

Transfer learning and kernel based multitask learning

I have a number of papers on transfer learning that I have started looking through for one of my research projects. Transfer learning is a machine learning technique where the basic idea is to learn multiple related tasks simultaneously, rather than learn each task independently. Previous work have shown that this works better, especially for predictive performance, compared to independent, single task learning. For my problem, I am interested in using multitask learning techniques based on natural extensions of kernel based methods for single task learning, such as the Support Vector Machines (SVMs).

Go programming language

One of my friends from the compiler group sent me an email regarding Google's new programming language, Go. From the Google talk presentation it seems this language has some unique and amazing features, and most interestingly, it feels like a language where one can code like python and get power of C++. Check out this talk if you want to know more about the language.

Compressive sensing, again

I am currently reading a paper titled 'Random projections of smooth manifolds' by Baraniuk and Wakin. From a machine learning point of view, this paper is very interesting since it deals with dimensionality reduction, a very useful technique in machine learning research. The other great aspect of this paper is its close relation to compressive sensing (cs) ideas, a topic that I have been excited about for a while.

Computational tools for cooking

You don't need to be surprised reading the title, it does put Computational tools and cooking together in the same sentence. One of the perks of being a new graduate student is cooking for oneself, or rather, as in my case, learning to cook for oneself. It aptly fits in with my area of interest, machine learning, since I am learning to cook, identifying patterns in cooking from my training experience, and using those ideas to further refine my cooking to perfection.

However, I am not referring to myself by the term Computational tools in the title. Besides all the traditional tools necessary for cooking, my list involves a laptop, webcam, and internet. Honestly, I don't have a good idea of the quantity of spices to use or how to cook certain food items yet, so I take help of Google to find out recipes and figure out how much of each spices to use, if any. It is not as simple as searching for a recipe though; most times I don't have all the necessary ingredients mentioned in the recipes that I find, so I improvise by abstracting information by combining multiple recipes and some experimentation. This technique has served me adequately so far.

In the extreme case, which, very fortunately, has not happened yet, I have no idea of how to go about cooking something, skype and video chat seems like a great resource that will come in handy. Just call up someone who knows how to cook, and use the cam to show what disaster you have been up to and follow directions from that person. One great advantage of this telecooking technique is that you are not limited by geography, and can potentially go international; except that time zone might be a contributing factor.

Graph theory and probabilistic inference

Graphical models and probabilistic inference have become an intimate part of current machine learning research. I came across a paper by Professor Michael Jordan that gives a great introduction to probabilistic inference techniques based on graphs. Here is a link to the paper.

Keys to my office space

I got the keys to my office and access card to the cs building today. After picking up the keys and card, I went into the office to claim my desk. I got a nice, lighted space on the corner, with a set of lock and key drawers as well. I am feeling pretty excited having my own office space, a sanctuary to delve into deeper thoughts and understand the intricacies of computer science. It will be a while before I become a regular at my office, but I am looking forward to it already.

Law of brain activity and temperature

I think this law is generally true, $\textstyle \text{level of brain activity in research} \propto e^{-(\text{outside temperature})}$ Please don't quote me on this.

California heat...research

I have been back a few days only, and the heat has already started to kill me. It has made me listless, not that I wasn't before, and almost sucked out all the energy in me. Amidst all this, I met my advisor today to discuss a research problem to start on. Since classes don't start till later, I can use the time in between to, hopefully, take a few stabs at the problem. Let me try to summarize the general research problem,

Use some form of probabilistic machine learning model to tweak segmentation/recognition of objects in natural images using high level knowledge (forming the ground truth).

The summary is very vague, but I will try to give exact details in later blog entries.

My books wishlist

I am adding these two books to my wishlist. From what I have read in the reviews, both of these are going to be an enjoyable read.

• Society of Mind by Marvin Minsky.
• Godel, Escher, Bach: An Eternal Golden Braid by Douglas Hofstadter.

Actually, the second one was recommended by a college friend.