Observations on the current state of software development for Web applications

This article is intended to describe what I've observed over the past ten years on the evolution of programming for Web applications. My subjective perspective has been influenced by having returned to software development after an absence of a few years, such that I'll start off with a brief introduction to explain the context.

Setting the scene 

About 15 years ago, I took some time off from my software (s/w) development career to complete an MBA. At the time it also became very important to devote more attention to raising my younger kids for a while, since their mom needed to focus on a critical period in her own career.

Hence I balanced my part-time studies with parenting, until finishing the MBA in 2007, after which I gradually returned to working as a s/w engineer full-time. Initially this focus on hands-on development was primarily intended to reestablish my programming skills, which had become rusty.

However, in the course of this process I rediscovered my love of programming, which (like many people) I'd previously set aside in favor of management.

At first it was hard to get back into the groove, but eventually I hit my stride.Thus, over time I’ve been able to hone my previous programming skills and learn enough new ones to embark on the development of a new product.

This new venture is happening via a start-up company called “Simulation Magic”, but more on that later. For now I’d like to reflect a bit on what I’ve observed in the s/w world, since returning to “active duty”.

Summarizing my own areas of expertise

Just to give you an idea of where I’m coming from, over the past ten years I’ve found my niche by gradually specializing in the combination of a few specific areas, as follows:

·        Primarily I’ve worked with the Microsoft .Net platform, either to develop and maintain Web applications, or to convert legacy Windows and Unix apps to the Web

·        For code development, I’ve used mostly C#, HTML, CSS, Visual Basic (VB) and JavaScript, but lately I’ve used Angular JS a lot too

·        From the database (DB) side, it’s been mostly SQL Server, with a fair amount of report development via SQL Server Report Server (SSRS)

·        Along the way I’ve also become reasonably adept at using Generics, which are now such an essential part of contemporary object-oriented (OO) development environments.

Meanwhile I’ve also dabbled in the use of some IBM tools, like Web Sphere and DB2, as well as doing some s/w development with Java and Linux-based platforms. Before going back to school to do my MBA, I’d been programming for many years with a variety of languages, from Fortran, to Assembler, to C++.

Over the course of my career, I’ve also worked on the development of diverse s/w products, from embedded firmware, to healthcare management, to banking applications, to process control systems.  My educational background includes several university degrees, with specializations in math, computer science, engineering and finance/accounting.  

At the same time, I’ve also come to enjoy the challenges of “forensic software”, which from my point of view includes anything that pertains to digging deep into a given set of program files. Thus what’s interesting for me in this area is the process of systematically identifying and resolving bugs, as well as refactoring existing code (i.e. as opposed to the type of forensic work that's typically done by security-oriented institutions).    

All of that led me to my latest venture, which is a new s/w product that combines Big Data, Analytics and Simulations. In brief, the idea behind the "big data and analytics" phase is to start by assimilating and describing the diverse input data from various sources. Then the next step is to analyze it, in order to predict the likely outcomes.

Thus the subsequent “simulation” phase is intended to illustrate all possible scenarios, in order to use mathematical forecasting to prescribe ways to optimize those outcomes. Thus the system is intended to thereby both facilitate decision management and mitigate risk.   

Further details on this approach to Big Data, Analytics and Simulations were provided in my previous posts to this blog, such that I won’t elaborate here. However, to me all of this is essentially a way to make use of the diverse skills that I’ve accumulated over the years.

Another factor, though, is that it’s a way to avoid the biggest enemies of typical s/w developer/nerds like me: complacency and boredom.

How developing s/w is analogous to renovating a house

One of my hobbies is renovation, such that I’ve had the opportunity to be involved in the renovation of a few small homes and rental properties over the years.  However, since renovation is strictly a part-time gig for me, this often involves hiring contractors to get the work done.

Thus a valuable rule of thumb for this activity is that I try to avoid asking anyone to do something that I wouldn’t know how to do myself. Of course, sometimes this may just mean ensuring that I do the necessary research first, since there are many aspects of renovation that I either don’t have practical experience with or don’t wish to learn about (e.g. I’m lousy when it comes to plumbing).     

Well, in some ways it’s the same thing when it comes to s/w development: after all, how can you be an effective team manager if you ask people to do stuff that you don’t know how to do yourself? As with home renovations, if you sometimes don’t know how to do a particular programming or database task then that’s okay, since s/w development is so incredibly diverse today.

Game changers

Nevertheless, you can still:

·         Read a relevant book, or Google the topic, so that you can at least talk about it intelligently

·         Admit to your team that you haven’t done this particular thing yourself before and then either learn it on your own, or have the humility to learn it alongside them

By employing this approach with respect to renovation work, I'm able to quickly smell a rat when contractors try to pull a fast one on me, such that I can then unceremoniously get rid of them. Conversely, when someone does a great job, then I can be suitably impressed and express my sincere appreciation for their work. 

In fact, I have great admiration for anybody who is able to display demonstrable expertise and/or craftsmanship in just about anything. So, if there’s one thing that I’ve learned, it’s that I need to shut up and listen to the experts. Hence, while I know enough to be dangerous in a lot of areas, I generally surround myself by people who know much more than I do … and then I gladly take their advice.

Thus the topics that come to mind, when I think of game-changing innovation for Web development, include the following:

·         The evolution of Generics

·         The maturation of Angular JS and its siblings

·         The convergence of Java and C#

·         The ubiquitous application of Artificial Intelligence (AI) to so many contemporary areas of s/w development.

Generics

Over the past few years, the use of Generics has played an increasingly important role in the evolution of object-oriented programming. For example, they make it possible to include anonymous method calls that will only get identified at run time, which is a very powerful feature. Another frequently-used application of Generics is the Lambda function call, which has become very common, particularly whenever we access a database using an object relational mapping (ORM).

Thus the DB calls tend to become more readable and less error-prone when we use Lambdas, since they appear to go directly to the entity that we seek, rather than systematically searching through the records for a match to our given query/conditions. However, I say that they "appear" to do this because this is actually an illusion and can thus be a double-edged sword.

In fact, those lovely and intuitive Lambda functions will ultimately get automatically translated into some more convoluted SQL code, which does indeed search the DB until a match either is or isn’t found.

Hence a trade-off occurs when increased clarity of using Lambdas is offset by the price of auto-generated SQL code, which typically won't be as optimal as that which would be written by an experienced SQL programmer.  Meanwhile - as anyone who has ever tried to debug a Lambda statement knows - stepping through this code can be challenging, as the contents of the Lambda function call will generally get treated as a single block by the debugger. 

Finally, there is the tricky issue of two-pass Lambda calls to deal with. This occurs because the return of the data and the execution of surrounding code loops do not occur in parallel. This is not intuitive and can be confusing, such that this process sometimes requires the use of nested loops

The result of this approach ensures that the data that's fetched from the DB by the Lambda call is actually present when the appropriate iteration of a surrounding code loop is executed. Similarly, the convenience of using anonymous methods is offset by the relative complexity of reading code that does not explicitly name the methods that are called.

So, perhaps the moral of the story is that (despite the appeal of Generics) there is still no free lunch.

Angular JS and its siblings

I’ll restrict my comments to Angular JS here (which I'll refer to simply as Angular), since it’s the most popular brand of its ilk, but this discussion applies equally to other similar tools. Ditto for Java vs C#, which I consider to be functionally equivalent as programming languages, even though one is compiled and the other is interpreted.

Before learning Angular, I’d already become fairly adept at using JavaScript (JS), in combination with old-style VB code.  Hence I was fascinated by the way that JS lets you identify and manipulate specific objects on the client side, once the appropriate handle has been found for a given object.

Since this already seemed magical to me, I was pleasantly surprised when I later learned that Angular takes this a step further, by providing a rich object-oriented environment that includes Generics. Having come from a background that's heavily oriented towards C#, I was amazed at how similar the client-side Angular programming experience is to the analogous server-side C# programming.

On the other hand, I get the sense that new developers who have gone straight to client-side programming (without doing server-side development first) have been saddled with a limited perception of how to optimize their code. It’s a bit like considering the case of C++ programmers who never used Assembler … which means that they may not be aware of the low-level implications of their relatively sophisticated C++ code.

So, when it comes to Angular vs C#, the problem (as I see it) is that there is an increasing tendency to try to do everything on the client side, rather than determining when to use server or client-side code on a case-by-case basis. Again, although Angular seems to magically access the DB (mostly via client-side Lambda function calls), generally this DB access will ultimately be automatically translated into server-side SQL code.      

Now don’t get me wrong – I realize why it’s more fun to do everything via client-side Angular, rather than sometimes resorting to those relatively tedious server-side methods.  However, from an optimization point of view, there is a significant performance hit when the automatically-translated server-side code is generated, instead of using manually-produced code that is optimized by an experienced server-side programmer.

Hence (I’m just saying), when performance is an issue, it’s usually worthwhile to actually do the triage of when we should use server vs client-side code.  After all, database access often seems to be the performance bottleneck in high-volume Web apps, which is one of the main reasons that we use distributed systems.  

Thus I would simply caution multi-disciplinary programming teams to distribute the load among themselves, just as the app itself distributes it among servers.  

Java vs C#

As alluded to above, it seems to me that there has been a convergence of Java and C# over the past ten years or so. Again, I realize that at a low level the interpreted code will behave very differently from compiled code, but from a developer’s perspective I think that this difference may often seem relatively transparent.

Hence my take on this is that Microsoft (MS) has striven to emulate the freedom that Java-based, open-source code provides. Of course, the open-source world still offers developers an unparalleled freedom of choice, when it comes to mixing and matching the tools of their development environment. 

Nevertheless, Microsoft's C#-based .Net development environment offers out-of-the-box access to virtually everything that a typical developer needs, which can be a big advantage in terms of “hitting the ground running” for a given project. Interestingly, the same seems to be true of other modern programming languages that are supported by the .Net environment, such as VB, R, S, Python and so on.

The application of AI to mainstream s/w development

Thus that Segway leads me neatly to the increasing use of AI (and the related programming languages) in everyday s/w development. It seems reasonable to assume that this phenomenon is occurring largely because we’re witnessing a significant rise in the use of Big Data and Analytics for typical Web applications.

This approach makes sense, given that we’re now in a world where our clients are increasingly seeking a competitive edge, such as that which can be obtained through the use of automated trend-analysis. In particular, it’s normal that corporate strategists want to exploit the available customer-based data, in order to determine how to effectively focus their marketing efforts ("like a laser beam") on their targeted audience.

Consumers tend to feel that this may be an invasion of privacy, since it can be somewhat disconcerting to see an obviously targeted ad appear when we’re browsing the Web in a seemingly anonymous mode. Naturally, Web-surfers can take measures to prevent this from happening (e.g. by disabling cookies and so on), but it appears that most people won’t bother.

Conversely, anyone who is selling something generally needs to seek a competitive advantage ... and the use of Analytics seems to be the contemporary ticket for achieving this goal. That’s in part why I’m turning my own attention in that direction, in order to set my sights on the application of AI to everyday s/w development, as described in my earlier blogs.

Personally, I find it amazing that we can use Analytics to analyze Big Data, in order to:

1. Intuitively describe the situation that's represented by the data, so that we can 
2. Predict the relevant confidence intervals for likely outcomes and then
3. Prescribe appropriate strategies, to thereby increase the likelihood of the most desirable results occurring.

From my point of view, all of this is relevant to a plethora of applications, from the optimization of manufacturing processes, to medical diagnoses, to healthcare management, to finance, to sales and marketing.

Hence.while the engineer in me loves the way that we can apply mathematical algorithms to everyday problems, the computer scientist enjoys the cool programs that this entails. Meantime, the businessman in me drools over the potential revenue.

So, going forward I’ll be providing more info on evolution of this product via a related blog that can be found at www.SimulationMagic.com (i.e. it’s currently under construction). Meanwhile I’ll continue to be working primarily in general s/w development for the Web, since that's still my bread and butter.

Nevertheless, I'm looking forward to gradually becoming more involved in this new adventure, by applying AI to the automated support of decision management and the mitigation of risk.          

Benefitting from the combination of Analytics, Big Data, Cloud and Multi-Platform development

According to Gartner Inc., one the world's leading information technology research and advisory companies, the top 10 fastest growing strategic technology trends include Analytics, Big Data, Cloud and Mobile applications. To the latter might be added Multi-Platform support, since users may want to view your Web applications anywhere, at any time. [1]

However, trends come and go, without necessarily having practical significance. Can businesses benefit from the combination of these technologies? The answer is yes and here’s a brief step-by-step explanation of how. We’ll start with the definition of Analytics.

The three aspects of Analytics

There are actually three aspects to Analytics: descriptive, predictive and prescriptive. The descriptive part involves illustrating the meaning of data. For example, this could mean displaying an explanatory message (text), sets of rows and columns (tables), bar charts, histograms, etc. (graphs). 

More advanced programs will depict the information using sophisticated 2D and 3D graphics, such as the realistic rendition of a cockpit in a flight training simulator. This would be analogous to the simulations that are provided in games, but in this case they would be intended to accurately represent reality, rather than some imaginary world.

The next step in Analytics is to predict the likelihood of a given set of possible outcomes, by forecasting the results for various scenarios. This is common in finance, but it is also applicable to simulations of equipment, technical training and marketing situations (e.g. depicting a given customer in a specific scenario, such as buying a condominium or a car).     

Finally, there is prescriptive Analytics, which involves recommending some prescribed action, based on the above results. In other words, you must ask: “How can my client save money, or otherwise benefit from the above predictions?”  

Therefore this is the key to profiting from Analytics … it is the value-added feature that accompanies the treatment of data in fields where information overload is common (e.g. energy management, financial risk management, healthcare management, etc.).

Applying Analytics to Big Data

             

This brings us to the term Big Data, which is bandied about freely these days, although not everybody who uses it actually knows what it means. So, what is it and what is its relationship to Analytics? The key here is information overload: there is simply too much data being collected for most applications, such that it becomes a challenge just to make sense of it.

Thus the treatment of Big Data involves extracting what is relevant to the user, much like the objective of Analytics. However, in the case of Big Data the amount of data is huge, such that the application of Analytics is intended to both display and analyze a relevant portion of that data coherently.

In particular, a given outcome can be predicted with a specific confidence interval (e.g. “there is a 76% likelihood that this unit will fail within 2 weeks”) and remedial action can be prescribed accordingly.  Thus the 3 aspects of Analytics can be applied to the Big Data, in order to recognize, predict and correct problems as they occur.       

Deploying Cloud applications to treat Big Data using Analytics

Typically the above scenarios are addressed via a Web-based Portal or Dashboard display, which allows registered users to monitor their systems and alerts them if there is an unusual situation.  This usually involves a customizable graphic user interface (GUI), which can be divided up into parts that correspond to the specific points of interest for users.

For example, critical data might include:

- Operating parameters of HVAC equipment for an energy management application

- Fuel consumption for a fleet of vehicles or

- Physiological measurements for a medical monitoring system

All of these systems might be accessed from a central server that provides customized Portals or Dashboards for a given set of registered users. This is, by definition, a cloud application and it is increasingly the preferred choice for deploying robust Web applications.

Dealing with government/industry Regulations and Security

After these users log on securely, they should be presented with the specific data that is of interest to them, in a format that makes sense. In other words, they are not expected to waste their time trying to understand superfluous information that is not relevant to their goals.

As a result, the treatment of Big Data via Analytics usually requires considerable subject matter expertise, which may or may not be regulated by government authorities.

This information is typically sensitive, such that measures to deal with security and encryption then become necessary. Furthermore, the presence of large numbers of simultaneous users will require the system to be both scalable and optimized for performance.

Finally, in addition to accessing the information that is available on their personalized Portal or Dashboard via a browser, users will also likely want access to this via a mobile device or tablet instead.

Developing mobile and platform-independent applications

In order to make these applications accessible on either mobile or other platforms, they must be developed with multi-platform use in mind. With a Cloud-based application, the server-side software becomes transparent to the user, who can access it via a browser on a PC or Mac, as well as a tablet or mobile device.

Thus the server can run the development environment (be it Microsoft C# or Open Source Java) and the operating system (such as Linux or Windows) of choice. Hence tools like Knockout GS and Kendo UI can be used to make the applications portable, in order to ensure that each user’s platform is identified and the available screen space is optimized.

Designing the application with a Model View Controller (MVC) paradigm helps ensure that there is a separation of concerns for the development team. In other words, the various developers on the team might focus exclusively on the client side, back-end or middle-ware.

Fine-tuning the user interface

Development of the GUI can be made even more granular, by using the Model-View-View-Model (MVVM) approach, which initially grew out of the Windows Presentation Foundation (WPF) initiative. This approach further breaks the GUI layer down into its own model, view and controller paradigms.

The use of specialized Java Script tools (such as JQuery and Ajax) allows further fine-tuning of the client-side software, thereby diminishing the load on the server and enhancing the response time for end users. For example, JQuery allows “accordion” type screen sections that dynamically select only relevant blocks of information for display on the screen.

Similarly, Ajax can be used to dynamically refresh only specific sections of the display, as required, thereby avoiding the need to annoy users by frequently refreshing the entire page. The result is that the user can experience roughly the same kind of performance on whatever platform he or she is using at any given time.

Aiming for a satisfying user experience

Thus the bottom line is that users must feel that their time is optimized and that they are as comfortable as possible with the application. For example, a surgeon checking a patient’s status while golfing on his day off can access the same information on his smart phone that he’d see on his PC or tablet at work.  

Similarly, a salesperson in the field can use her smart phone to access the same mission-critical information that she’d have at her desk, back in the office. By the same token, the system should be intuitive enough that new users can benefit from it as soon as possible, without the need for a steep learning curve.

That is a critical factor for sales as well, since your sales force will need access to easily understandable and demonstrable prototypes of the system, in order to show it to potential or existing clients. Whatever the circumstances, the use of these technologies and development strategies can result in significantly increased profits, by providing your customers with value-added features that zero in on precisely what they need to know.  

----------------------

[1] From: apmdigest.com/gartner-top-10-strategic-technology-trends-for-2013-big-data-cloud-analytics-and-mobile 

How to analyze a software application for weaknesses and come up with a plan to stabilize it?

Since this is a very wide-ranging question, I will narrow it down to the range of mobile applications that are most relevant to the example chosen here, which is a mobile medical records application. To start with, let’s consider the following scenario:

a)   A tablet-based application with a Windows-based user interface.

This is essentially a Windows application, but which allows the user the convenience of a pen- based interface. Let’s assume that the pen based interface works fine. Then, from a technical perspective, the present analysis can mostly focus on whether the application meets the user’s business requirements and on how well it performs.

Firstly, the performance is irrelevant if the application does not meet the needs of the user! Therefore, here are some questions to ponder:

•     Does the weakness that we are considering stem from an inability to meet these requirements?

•     Do we have clearly documented business and/or functional requirements to refer back to?

•     Was the correlation between requirements and functionality meticulously tracked during analysis, design and development?

•     Do we have documented Use Cases to explain the appropriate role playing for this application?

If we did our jobs properly in the first place, then the answer to all of these questions is “yes” and then we can simply match up the actual functionality to the business requirements. This should allow us to determine where the weakness lies and how to address it.

However, to make this more interesting, let’s assume now that we’re dealing with a legacy system that was ported by the company to a mobile application. In that case, we may be dealing with a system that has no proper documentation to refer to.

Here is where proper quality assurance comes into play! Since we’re assuming that the normal process of rigorous testing was never done on the original app, it becomes incumbent upon our intrepid QA person/people to test the heck out of it, in order to determine where it breaks.

That’s the first step in pinpointing where the app is weak, if we have no specs to refer back to. If we do have specs then the task is much easier, but the principle of using rigorous testing to determine when it breaks still applies.

Likewise, if feedback from the users was not part of the development process (which it would have been if we using an Agile methodology!!!), then it is essential to get that feedback now. In other words, if people are already using the app, then we need to know exactly when/how it does or does not meet their needs.

Again, I must emphasize that this should have happened before the app was released, but failing that, let’s get this input now. This will lead to a set of functional requirements that can then be compared with the test results for our QA staff. Essentially the take home message here is this: a combined effort by the analysts, designers, developers and QA staff results in an application that has no surprises. Ideally this would happen before release, but better late than never!

Once all of this has been done, then we can fix the bugs and/or enhance the product, as needed. At this point we know what it actually does versus what it should do, such that it is now a straightforward process to determine what needs to be done in order to stabilize it (i.e. it’s understood that we’re including bug tracking as a subset of this whole process).

Of course, now we come to the ubiquitous issues of maintenance, such as:

•     How long will it take?

•     How much will it cost?

•     Who will do the work?

•     Who will pay for it/

•     When will it be done?

Sorry to sound like broken record, but once again Agile development comes to the rescue here. After all, fixing or enhancing an app is just a microcosm of the original, overall development process. Thus, the same principles apply:

•     Address any unknown/risky issues first

•     Get constant feedback from the users

•     Ensure effective communication within the dev team, to identify/address problems early

•     Release small, frequent iterations

•     Test constantly!

b)   How corporate strategy directly affects this situation

Another important consideration here is, “What are the relevant apps of our competitors doing?” Thus the above process should also take into account whatever the competition currently does and whether our product is up to snuff.

Indeed, it may even be that the competition has recently added features that make the latest release of our app obsolete! In that case, our app may work just fine and yet still have a significant weakness. Similarly, the technology behind the app must constantly be re-evaluated, with respect to what is currently available “out there”.

The best designed and implemented app in the world may become useless if the technology is

no longer appropriate (e.g. the Yellow Pages). So, this suggests that s/w development should not stand alone: marketing, strategy, product management and other factors need to be considered as well.

The most effective way to ensure that this occurs is to create a system of overlapping competencies, so that all the bases are covered by more than one individual (i.e. like baseball players backing each other up on the field). This requires us to foster an atmosphere of “covering each other’s backs”, as opposed to “looking over each other’s shoulders”. There’s a subtle but critical difference between the two approaches!

Hence, for example, we should have the product manager involved in quality assurance, the development team leader concerned about business management, the chief technology officer wondering about corporate strategy, and so on. Since that appears to be the case already, then the challenge is to propagate this mentality going forward, as we grow.

This leads to two further questions, right off the bat:

•     How do we maintain the sense of family/mutual assistance as we grow from a small to medium to large company?

•     Why hasn’t this company grown more since its inception?

These are tough questions that are best addressed when we meet. However, I will say this here: if we can’t maintain a congenial atmosphere, then the issue of growth becomes moot (unless we want to become just another big, impersonal corporation).

Conversely, that growth may have been inhibited (intentionally or otherwise) by a desire to maintain that atmosphere. In that case, since our products and the related market are both clearly conducive to growth, then the challenge is clear (which is where I come in!).

c)   Addressing the above scenario with apps developed for more specialized devices, such as the

Blackberry, iPhone or iPad

This issue is also beyond the scope of the present discussion and thus best addressed when we meet, but a few quick notes are appropriate here:

•     How do we address potential problems with proprietary software? By the way, this also applies to the use of products from potential partners (e.g. Procura, Qualicode and Motion Computing).

•     What tools are available to diagnose potential hardware problems that can affect our software products? As trusted partners of RIM, do we have complete access to their diagnostic tools? Similarly, what if the initial assumption above was invalid and the pen based interface was actually the problem – what tools does Motion Computing provide for analyzing their pad computers?

•     How relevant is the user’s perspective? Since our products are intended for various user categories (agencies, home care nurses, personal support workers), then the weaknesses in a given app may depend largely on the user’s role.

•     Is language an issue? By developing multilingual products, we expose ourselves to potential glitches that derive from the improper mapping of database objects to the application code. On the other hand, if we did/do this right, then we can just as easily accommodate ten languages as two (i.e. the relational database design should make the mapping for numerous languages as simple as for two). Qualicode is now looking to the European market – are we? What about Asia? 1

•     Why are we not developing apps for Apple products? Unless this is prohibited by one

of our partners (e.g. RIM), then what would be involved in this? Would all of the above apply if we were to develop apps for Apple products as well?

•     What is the meaning of life?

Okay, maybe that last one was just a bit of a stretch … but the answers to these and many more questions will be forthcoming over the course of our upcoming meeting(s). I’ll be that you just can’t wait!

1 Actually, it occurs to me that the 7 languages which I either speak or understand are all European … so, if you want to wait a bit to address the Asian market (until I get a chance to learn Mandarin also), then that’s okay.