The impact of micro-services and containers on application and enterprise architecture - Part I

Historical paradigm shifts in programming

Since its inception in the mid-20^th century, the field of software development has undergone several paradigm shifts. For example, the introduction of structured programming concepts in early coding languages (such as Fortran, Pascal and C) made the resulting code far more manageable than it had been originally.

Later, the advent of object-oriented programming techniques allowed encapsulation to be routinely applied in the next generation of languages, like C++, Smalltalk, Eiffel, etc. Similarly, the subsequent introduction of software design patterns added a significant level of sophistication to the practice of both application and enterprise architecture.

The resulting paradigm shift could be likened to:

• Replacing “spaghetti” code (i.e. tangled modules with no structure or encapsulation)
• By “lasagna” code (i.e. layered modules, with data abstraction and information hiding)

As desktop apps gradually became ubiquitous, various types of Windows forms were introduced to develop the interfaces for end users, for the interactions with databases and so on. Then, as the Internet became more pervasive, these were replaced by Web forms.

This became common in popular multi-purpose languages like C# and Java, as well as in tools intended for more specialized purposes (e.g. like Oracle Developer, for programming database applications). 

A common sense approach that led to implementing Agile development methodologies

Meantime, the advent of Agile development methodologies revolutionized the management of the software life cycle, by providing a viable alternative to the traditional “waterfall” model.  Over time, the progression of methodologies looked something like this:

The Waterfall method:  

• This approach involved the sequential development of large, highly coupled modules
• Thus the start of work on subsequent modules often depended on the completion of the previous ones

Extreme Programming:

• This approach made use of paired programming and aggressive testing
• The idea was to ensure the implementation of smaller modules that worked reliably and were easy to test, instead of huge monolithic blocks that were difficult to maintain
• This was achieved by having programmers work in pairs, so that they could constantly compare ideas and ensure adequate test coverage

Incremental development and deployment, with constant customer feedback:

• This is the essence of the “Agile” approach, which was an outgrowth of Extreme Programming
• It involved deploying apps after relatively short “Sprints”, which each included their own analysis, design, development, testing and deployment phases
• In other words, each Sprint became a microcosm of the entire software development life cycle
• In order to ensure constant feedback from clients (and resulting adjustments as needed), the length of Sprints were typically measured in weeks
• Hence this represents a huge change from the Waterfall model, where deployments were scheduled over periods of months or even years
• Meanwhile the typical inefficient, boring and excessively long weekly status meetings were replaced by short, daily “stand up” team meetings (generally referred to as “Scrums”)
• This ensured that developers always had timely feedback (and if necessary, help) from their peers
• The three daily questions became:

i. What did you do yesterday?

ii What will you do today?

iii. What are your blockers (i.e. where can you use some help)?

• Where possible, subject matter experts (SMEs) were embedded into the dev teams, to ensure that the code continued to conform to what the clients wanted

Hybrid approaches: 

• Evolving Agile methodologies have resulted in sub-specialties, as such as:
• “Lean” (i.e. minimal or just in time) development and
• “Scrum Masters” (i.e. the people who manage the Scums and Sprints) 
• By and large though, a hybrid approach to software development has emerged
• Therefore, companies today will often pick and choose whatever development methodology (or combination thereof) works best for them.

Continuous integration and automated database management leads to the evolution of Dev Ops

As a result of the evolving Agile approaches to software development, the risky practice of infrequent monolithic deployments has been gradually replaced by providing smaller periodic ones. Thus, highly coupled and monolithic legacy apps were gradually either refactored or replaced, by a series of decoupled “black box” modules, which only talked to each other via strict APIs.

Many specialized languages also began to emerge for various highly specific needs (e.g. SQL, PEARL, PHP, R, Python, Angular, React, etc.). Meanwhile, various object relational data mapping (ORM) tools made database programming much more accessible to programmers who were not familiar with longstanding SQL coding practices. Similarly, with the introduction of concepts like business-intelligence, data mining and Analytics, the administration of databases started to become more of an ongoing, automated integration process.

Thus the various data types from relational database, data warehouses, Data Lakes and so on all had to be combined and optimized. This was necessary so that the underlying data could be effectively harnessed and explored. Then, as dev ops came to be increasingly automated, the use of continuous testing, integration and deployment became the norm.

This resulted in test-driven development and a greater emphasis on quality control. Lately this approach has been tempered somewhat by the concept of minimal viable products (MVP), which ensure that “the perfect does not lead to the sacrifice of the good”. Nevertheless, the trend remains to modularize software as much as possible.

Further replacing monolithic applications by the introduction of modular “micro-services”

Another major paradigm shift is occurring, as software apps are being split up into increasingly smaller components, through the increasing use of micro-services, containers and orechestration. These components are designed to provide highly specialized services, which generally communicate via RESTful APIs or asynchronous message queues.

The advantage of breaking up an application into small, manageable components (i.e. micro-services) is that the resulting modules can be maintained, enhanced of replaced independently of each other. As long as the interface “contracts” between them are respected by their APIs, the inner workings of each micro-service can be fully encapsulated.

This result is true regardless of whether the modules communicate via so-called “RESTful” HTTPS service calls, or via other API paradigms, such as asynchronous message queues. The former type of service calls are becoming more common and typically involve an HTTPS Put, Get or Pull of some kind.

This generates a “response code”, which indicates the status of the HTTP request, along with some data (e.g. using JSON, YML, etc.), if applicable.

Asynchronous message queues are very useful when the services that are being provided require a waiting period. Hence the message queue is polled periodically to determine if the results of a given query are available, but in the meantime the calling programs can continue to go about their business.

This approach represents a significant step forward, since it hugely diminishes the risk of making changes to a given service. For example, if a given micro-service is negatively affected, then it simply goes offline until it’s fixed. This approach was pioneered by Netflix when it first introduced streaming video services, but it’s now common among Big Data users, like Google, Facebook, Amazon and so on.

The ensuing need for containers and orchestration

The resulting micro-service modules lend themselves naturally to being “containerized”, using tools such as Docker. In particular, this approach allows the micro-services to be easily instantiated, tested and refactored independently … as long as the underlying communication contracts between them (i.e. their APIs) are respected. [1]

In fact, as a result of the increasing implementation of micro-services, the use of containers and orchestration actually become essential (rather than optional) tools. In particular, “orchestration” refers to the process of managing the containers automatically, via tools such as Kubernetes. This introduces the next level of features, such as automated scalability and “self-healing” to the containerized micro-services (more on this later).

The key here is that micro-services involve numerous small modules communicating with each other via RESTful or other APIs. When these modules are implemented, they become virtually impossible to manage manually. Hence the need to implement them via nodes, within containers, as well as the accompanying need for orchestration of the nodes (i.e. they’re required, rather than optional).

This means that a tradeoff results between the convenience of being able to modify the containerized micro-service independently and the ensuing need for some automated way to manage them.

While containers are implemented virtually, unlike the “virtual machines” which preceded them, containers do not require their own operating system.  Hence multiple containers can share a single OS and thus they consume very little in the way of resources. They can be “spun up” or “spun down” (i.e. created or destroyed) effortlessly, making them ideal for continuous integration and automated testing.

This means that continuous integration tools (e.g. Jenkins) can be configured to automatically build, test and integrate the containers.  Databases can also be spun up or down in this way, by implementing tools such as SQL Server Integration Services (SSIS) via containers. For example, tools like SSIS can automate the management of scripts that are used to deploy and maintain databases.

In the case of persistent data, the state must be maintained, which is a problem for containers that are constantly being spun up or down. Hence “volumes” are instantiated, which seamlessly connect to the containers (i.e. unlike containers, the volumes tend to exist on real media, such as disks).

This allows the data to persist, even when the containers are regularly created and destroyed. As far as the container is concerned, the connection to the database is essentially the same, whether it’s stored in a container or a volume.

Implementing micro-services via cloud-native orchestrated containers

The process of deploying applications “to the cloud” is intended to ensure that additional resources and scalability are automatically invoked, on demand, by the providers of the “platform as a service” (PaaS).

Tasks such as these are generally managed by Dev Ops engineers, who are specialized in working with tools like Docker, Kubernetes, Jenkins, SSIS, Kafka and so on.  This, in turn, frees up software developers to focus on what they do best - namely maintaining and evolving their application code. 

As alluded to earlier, implementing numerous parallel micro-services requires orchestration, because it’s simply not realistic to manage multiple containers manually.  Thus the orchestration tools provide the Dev Ops engineers with the ability to precisely configure the apps that they deploy. As a result, the number of instantiations available for a specific micro-service (and at any given time) is adjusted automatically.

For example, if it’s determined (e.g. through metrics, instrumentation, etc.) that N containers are needed to address the demand for a given service, then Kubernetes will ensure that there are always N containers available. In other words, if one or more Docker containers fail, then Kubernetes will gradually replace them until the desired number is restored.  This process is called “self-healing”. [2]

Similarly, if the demand increases or decreases, then the number of containers will automatically increase or decrease (as needed). This approach optimizes the use of resources, since the number of containers at any given time will be no more or less than what is dictated by demand. 

---

[1] Hence the remainder of this article will explore how the advent of containerized micro-services requires orchestration, which results in cloud-native apps that feature automatic scaling and provisioning.

[2] In the remainder of the article we’ll use Docker and Kubernetes as examples of containers and orchestration tools, respectively, for convenience. However, these concepts generally apply equally to the similar products provided by other vendors.

Update on my latest activities

In this article, we'll consider some of the software development environments that I've worked with  lately. This is a precursor to the subsequent posts on the impact that microservices and containers have had on  software development.

The .Net tech stack

Over the past 13 years I’ve used the typical .Net tools extensively, both as a developer and an architect. For example, this includes C#, SQL Server, SQL Server Reporting Services, Visual Studio, TypeScript and Angular.

Most of my projects have been built on the .Net platform, such that C# has been the primary programming tool for business logic. Similarly, SQL Server was the principal RDBMS for these projects, although I’ve often sued other tools interchangeably (e.g Oracle).

Visual Studio has been the editor of choice, giving us access to tools, such as Team Foundation Server, which was replaced by Azure Dev Ops.  Thus, we could use the built-in tools to track projects using an Agile approach, within the TFS/Azure Dev Ops framework.

For user interfaces, I’ve used various versions of Java Script, such as TypeScript. However, my preference is Angular, because it lends itself well to object-oriented design and development, very much like C#.  

In all the cases referred to above, as an architected I’ve been able to share my hands o experience with developers, allowing them to understand nuances that I came across as a programmer. I still enjoy being hands on to help my team solve problems and navigate proofs of concepts for approaches that they are not familiar with.

Serverless applications using microservices

I’ve been using microservices for virtually all of my applications since around 2019. Ther most extensive use has been for Simulation Magic, as described on the web site at www.SimulationMagic.com .  In the various projects for which I’ve been the architect, we’ve used primarily Kubernetes to manage the containers and Docker to create the containers themselves.

However, variations on this theme include various tools from AWS, which essentially mimic the Kubernetes/Docker paradigm (see below).

From my perspective as an architect, the primary advantage of this approach is to spin up/destroy containers as needed and automate the connectivity between them. I also like how microservices can be sued to create virtual servers for testing proofs of concepts, load management, etc.  

Finally, I like the way that responsibilities are compartmentalized within containers. To me, this is simply the next step in the OO paradigm, which aims to ensure that processes are black boxes that communicate via input/output parameters. This allows us to easily replace components within any system that I’m working on, either to repair or improve them.

AWS as microservices

I worked with AWS last summer for IBM on a project for Immigration Canada. This involved creating the infrastructure for a new digital application process for Canadian passports. The AWS system was the foundation for this project, which required various types of connectivity between platforms for security, verification of identity using AI, saving applications, training an image recognition, system, etc.

We encountered some latency issues with AWS, with respect to loading of services that were called automatically. However, we worked directly with Amazon engineers to address these problems. As a senior solutions architect, I was responsible for ensuring that dev team members were able to continue working without being blocked by such issues.

NoSQL

In the past 5 years, I’ve used NoSQL in projects where we needed an alternative to traditional Relational Database Management Systems. This has been useful in cases where the data being stored is in a format that is not easily accessible in relational database (e.g. Blobs, images, etc.).  For example, in the digital passport project for Immigration Canada, we used NoSQL to store photographs required for training of image recognition system.

However, it’s worth noting that NoSQL is not always appropriate. In particular, when we’re accessing small blocks of data that can be stored in tables, a relational database (using SQL) is still best. For Simulation Magic, we use a combination of both, since our Big Data requires Blobs of public and private data,

This data must first be filtered and then converted into relational data that can be treated by calculation engines used for Analytics and Simulations.

Education and learning software

I’ve actually been involved with education-oriented software since the start of my work as a software engineer career, going back to the early days of my career. More recently, I’ve been designing and developing Simulation Magic, the AI-based system referred to above. This is a joint project with SOLSER Information Technology, which is based out of Mexico.

The system uses Analytics and Simulations to mitigate risk, by helping people make decisions, based on whatever information is available to them. There are many applications for this, such as managing the allocation of funds for various projects with resources competing for attention.

However, a major application is in education, where simulations can be used to create realistic scenarios that approximate real life situations. For example, this approach can be applied to any scenario where diagnostics are required.

This includes medical diagnoses, decisions by technicians, supply chain management, operations, etc.  We are presently developing various versions for the minimal viable product (MVP) that will be used by Beta users in real life environments where they need to simulate and analyze the decisions required in their everyday workplace environment (see also www.SimulationMagic.com).   

Mocking

This is an approach that has been an integral part of all of my projects for the past 5 to 10 years. Here is how defined on the Stack Overflow web site: 

"Mocking is primarily used in unit testing. An object under test may have dependencies on other (complex) objects. To isolate the behaviour of the object you want to test you replace the other objects by mocks that simulate the behaviour of the real objects. This is useful if the real objects are impractical to incorporate into the unit test. In short, mocking is creating objects that simulate the behaviour of real objects."   [1]

------------------------------------------------

References 
1. See https://stackoverflow.com/questions/2665812/what-is-mocking 
 

 

 

 

  

 

Observations on the current state of software development for Web applications

This article is intended to describe what I've observed over the past ten years on the evolution of programming for Web applications. My subjective perspective has been influenced by having returned to software development after an absence of a few years, such that I'll start off with a brief introduction to explain the context.

Setting the scene 

About 15 years ago, I took some time off from my software (s/w) development career to complete an MBA. At the time it also became very important to devote more attention to raising my younger kids for a while, since their mom needed to focus on a critical period in her own career.

Hence I balanced my part-time studies with parenting, until finishing the MBA in 2007, after which I gradually returned to working as a s/w engineer full-time. Initially this focus on hands-on development was primarily intended to reestablish my programming skills, which had become rusty.

However, in the course of this process I rediscovered my love of programming, which (like many people) I'd previously set aside in favor of management.

At first it was hard to get back into the groove, but eventually I hit my stride.Thus, over time I’ve been able to hone my previous programming skills and learn enough new ones to embark on the development of a new product.

This new venture is happening via a start-up company called “Simulation Magic”, but more on that later. For now I’d like to reflect a bit on what I’ve observed in the s/w world, since returning to “active duty”.

Summarizing my own areas of expertise

Just to give you an idea of where I’m coming from, over the past ten years I’ve found my niche by gradually specializing in the combination of a few specific areas, as follows:

·        Primarily I’ve worked with the Microsoft .Net platform, either to develop and maintain Web applications, or to convert legacy Windows and Unix apps to the Web

·        For code development, I’ve used mostly C#, HTML, CSS, Visual Basic (VB) and JavaScript, but lately I’ve used Angular JS a lot too

·        From the database (DB) side, it’s been mostly SQL Server, with a fair amount of report development via SQL Server Report Server (SSRS)

·        Along the way I’ve also become reasonably adept at using Generics, which are now such an essential part of contemporary object-oriented (OO) development environments.

Meanwhile I’ve also dabbled in the use of some IBM tools, like Web Sphere and DB2, as well as doing some s/w development with Java and Linux-based platforms. Before going back to school to do my MBA, I’d been programming for many years with a variety of languages, from Fortran, to Assembler, to C++.

Over the course of my career, I’ve also worked on the development of diverse s/w products, from embedded firmware, to healthcare management, to banking applications, to process control systems.  My educational background includes several university degrees, with specializations in math, computer science, engineering and finance/accounting.  

At the same time, I’ve also come to enjoy the challenges of “forensic software”, which from my point of view includes anything that pertains to digging deep into a given set of program files. Thus what’s interesting for me in this area is the process of systematically identifying and resolving bugs, as well as refactoring existing code (i.e. as opposed to the type of forensic work that's typically done by security-oriented institutions).    

All of that led me to my latest venture, which is a new s/w product that combines Big Data, Analytics and Simulations. In brief, the idea behind the "big data and analytics" phase is to start by assimilating and describing the diverse input data from various sources. Then the next step is to analyze it, in order to predict the likely outcomes.

Thus the subsequent “simulation” phase is intended to illustrate all possible scenarios, in order to use mathematical forecasting to prescribe ways to optimize those outcomes. Thus the system is intended to thereby both facilitate decision management and mitigate risk.   

Further details on this approach to Big Data, Analytics and Simulations were provided in my previous posts to this blog, such that I won’t elaborate here. However, to me all of this is essentially a way to make use of the diverse skills that I’ve accumulated over the years.

Another factor, though, is that it’s a way to avoid the biggest enemies of typical s/w developer/nerds like me: complacency and boredom.

How developing s/w is analogous to renovating a house

One of my hobbies is renovation, such that I’ve had the opportunity to be involved in the renovation of a few small homes and rental properties over the years.  However, since renovation is strictly a part-time gig for me, this often involves hiring contractors to get the work done.

Thus a valuable rule of thumb for this activity is that I try to avoid asking anyone to do something that I wouldn’t know how to do myself. Of course, sometimes this may just mean ensuring that I do the necessary research first, since there are many aspects of renovation that I either don’t have practical experience with or don’t wish to learn about (e.g. I’m lousy when it comes to plumbing).     

Well, in some ways it’s the same thing when it comes to s/w development: after all, how can you be an effective team manager if you ask people to do stuff that you don’t know how to do yourself? As with home renovations, if you sometimes don’t know how to do a particular programming or database task then that’s okay, since s/w development is so incredibly diverse today.

Game changers

Nevertheless, you can still:

·         Read a relevant book, or Google the topic, so that you can at least talk about it intelligently

·         Admit to your team that you haven’t done this particular thing yourself before and then either learn it on your own, or have the humility to learn it alongside them

By employing this approach with respect to renovation work, I'm able to quickly smell a rat when contractors try to pull a fast one on me, such that I can then unceremoniously get rid of them. Conversely, when someone does a great job, then I can be suitably impressed and express my sincere appreciation for their work. 

In fact, I have great admiration for anybody who is able to display demonstrable expertise and/or craftsmanship in just about anything. So, if there’s one thing that I’ve learned, it’s that I need to shut up and listen to the experts. Hence, while I know enough to be dangerous in a lot of areas, I generally surround myself by people who know much more than I do … and then I gladly take their advice.

Thus the topics that come to mind, when I think of game-changing innovation for Web development, include the following:

·         The evolution of Generics

·         The maturation of Angular JS and its siblings

·         The convergence of Java and C#

·         The ubiquitous application of Artificial Intelligence (AI) to so many contemporary areas of s/w development.

Generics

Over the past few years, the use of Generics has played an increasingly important role in the evolution of object-oriented programming. For example, they make it possible to include anonymous method calls that will only get identified at run time, which is a very powerful feature. Another frequently-used application of Generics is the Lambda function call, which has become very common, particularly whenever we access a database using an object relational mapping (ORM).

Thus the DB calls tend to become more readable and less error-prone when we use Lambdas, since they appear to go directly to the entity that we seek, rather than systematically searching through the records for a match to our given query/conditions. However, I say that they "appear" to do this because this is actually an illusion and can thus be a double-edged sword.

In fact, those lovely and intuitive Lambda functions will ultimately get automatically translated into some more convoluted SQL code, which does indeed search the DB until a match either is or isn’t found.

Hence a trade-off occurs when increased clarity of using Lambdas is offset by the price of auto-generated SQL code, which typically won't be as optimal as that which would be written by an experienced SQL programmer.  Meanwhile - as anyone who has ever tried to debug a Lambda statement knows - stepping through this code can be challenging, as the contents of the Lambda function call will generally get treated as a single block by the debugger. 

Finally, there is the tricky issue of two-pass Lambda calls to deal with. This occurs because the return of the data and the execution of surrounding code loops do not occur in parallel. This is not intuitive and can be confusing, such that this process sometimes requires the use of nested loops

The result of this approach ensures that the data that's fetched from the DB by the Lambda call is actually present when the appropriate iteration of a surrounding code loop is executed. Similarly, the convenience of using anonymous methods is offset by the relative complexity of reading code that does not explicitly name the methods that are called.

So, perhaps the moral of the story is that (despite the appeal of Generics) there is still no free lunch.

Angular JS and its siblings

I’ll restrict my comments to Angular JS here (which I'll refer to simply as Angular), since it’s the most popular brand of its ilk, but this discussion applies equally to other similar tools. Ditto for Java vs C#, which I consider to be functionally equivalent as programming languages, even though one is compiled and the other is interpreted.

Before learning Angular, I’d already become fairly adept at using JavaScript (JS), in combination with old-style VB code.  Hence I was fascinated by the way that JS lets you identify and manipulate specific objects on the client side, once the appropriate handle has been found for a given object.

Since this already seemed magical to me, I was pleasantly surprised when I later learned that Angular takes this a step further, by providing a rich object-oriented environment that includes Generics. Having come from a background that's heavily oriented towards C#, I was amazed at how similar the client-side Angular programming experience is to the analogous server-side C# programming.

On the other hand, I get the sense that new developers who have gone straight to client-side programming (without doing server-side development first) have been saddled with a limited perception of how to optimize their code. It’s a bit like considering the case of C++ programmers who never used Assembler … which means that they may not be aware of the low-level implications of their relatively sophisticated C++ code.

So, when it comes to Angular vs C#, the problem (as I see it) is that there is an increasing tendency to try to do everything on the client side, rather than determining when to use server or client-side code on a case-by-case basis. Again, although Angular seems to magically access the DB (mostly via client-side Lambda function calls), generally this DB access will ultimately be automatically translated into server-side SQL code.      

Now don’t get me wrong – I realize why it’s more fun to do everything via client-side Angular, rather than sometimes resorting to those relatively tedious server-side methods.  However, from an optimization point of view, there is a significant performance hit when the automatically-translated server-side code is generated, instead of using manually-produced code that is optimized by an experienced server-side programmer.

Hence (I’m just saying), when performance is an issue, it’s usually worthwhile to actually do the triage of when we should use server vs client-side code.  After all, database access often seems to be the performance bottleneck in high-volume Web apps, which is one of the main reasons that we use distributed systems.  

Thus I would simply caution multi-disciplinary programming teams to distribute the load among themselves, just as the app itself distributes it among servers.  

Java vs C#

As alluded to above, it seems to me that there has been a convergence of Java and C# over the past ten years or so. Again, I realize that at a low level the interpreted code will behave very differently from compiled code, but from a developer’s perspective I think that this difference may often seem relatively transparent.

Hence my take on this is that Microsoft (MS) has striven to emulate the freedom that Java-based, open-source code provides. Of course, the open-source world still offers developers an unparalleled freedom of choice, when it comes to mixing and matching the tools of their development environment. 

Nevertheless, Microsoft's C#-based .Net development environment offers out-of-the-box access to virtually everything that a typical developer needs, which can be a big advantage in terms of “hitting the ground running” for a given project. Interestingly, the same seems to be true of other modern programming languages that are supported by the .Net environment, such as VB, R, S, Python and so on.

The application of AI to mainstream s/w development

Thus that Segway leads me neatly to the increasing use of AI (and the related programming languages) in everyday s/w development. It seems reasonable to assume that this phenomenon is occurring largely because we’re witnessing a significant rise in the use of Big Data and Analytics for typical Web applications.

This approach makes sense, given that we’re now in a world where our clients are increasingly seeking a competitive edge, such as that which can be obtained through the use of automated trend-analysis. In particular, it’s normal that corporate strategists want to exploit the available customer-based data, in order to determine how to effectively focus their marketing efforts ("like a laser beam") on their targeted audience.

Consumers tend to feel that this may be an invasion of privacy, since it can be somewhat disconcerting to see an obviously targeted ad appear when we’re browsing the Web in a seemingly anonymous mode. Naturally, Web-surfers can take measures to prevent this from happening (e.g. by disabling cookies and so on), but it appears that most people won’t bother.

Conversely, anyone who is selling something generally needs to seek a competitive advantage ... and the use of Analytics seems to be the contemporary ticket for achieving this goal. That’s in part why I’m turning my own attention in that direction, in order to set my sights on the application of AI to everyday s/w development, as described in my earlier blogs.

Personally, I find it amazing that we can use Analytics to analyze Big Data, in order to:

1. Intuitively describe the situation that's represented by the data, so that we can 
2. Predict the relevant confidence intervals for likely outcomes and then
3. Prescribe appropriate strategies, to thereby increase the likelihood of the most desirable results occurring.

From my point of view, all of this is relevant to a plethora of applications, from the optimization of manufacturing processes, to medical diagnoses, to healthcare management, to finance, to sales and marketing.

Hence.while the engineer in me loves the way that we can apply mathematical algorithms to everyday problems, the computer scientist enjoys the cool programs that this entails. Meantime, the businessman in me drools over the potential revenue.

So, going forward I’ll be providing more info on evolution of this product via a related blog that can be found at www.SimulationMagic.com (i.e. it’s currently under construction). Meanwhile I’ll continue to be working primarily in general s/w development for the Web, since that's still my bread and butter.

Nevertheless, I'm looking forward to gradually becoming more involved in this new adventure, by applying AI to the automated support of decision management and the mitigation of risk.          

Benefitting from the combination of Analytics, Big Data, Cloud and Multi-Platform development

According to Gartner Inc., one the world's leading information technology research and advisory companies, the top 10 fastest growing strategic technology trends include Analytics, Big Data, Cloud and Mobile applications. To the latter might be added Multi-Platform support, since users may want to view your Web applications anywhere, at any time. [1]

However, trends come and go, without necessarily having practical significance. Can businesses benefit from the combination of these technologies? The answer is yes and here’s a brief step-by-step explanation of how. We’ll start with the definition of Analytics.

The three aspects of Analytics

There are actually three aspects to Analytics: descriptive, predictive and prescriptive. The descriptive part involves illustrating the meaning of data. For example, this could mean displaying an explanatory message (text), sets of rows and columns (tables), bar charts, histograms, etc. (graphs). 

More advanced programs will depict the information using sophisticated 2D and 3D graphics, such as the realistic rendition of a cockpit in a flight training simulator. This would be analogous to the simulations that are provided in games, but in this case they would be intended to accurately represent reality, rather than some imaginary world.

The next step in Analytics is to predict the likelihood of a given set of possible outcomes, by forecasting the results for various scenarios. This is common in finance, but it is also applicable to simulations of equipment, technical training and marketing situations (e.g. depicting a given customer in a specific scenario, such as buying a condominium or a car).     

Finally, there is prescriptive Analytics, which involves recommending some prescribed action, based on the above results. In other words, you must ask: “How can my client save money, or otherwise benefit from the above predictions?”  

Therefore this is the key to profiting from Analytics … it is the value-added feature that accompanies the treatment of data in fields where information overload is common (e.g. energy management, financial risk management, healthcare management, etc.).

Applying Analytics to Big Data

             

This brings us to the term Big Data, which is bandied about freely these days, although not everybody who uses it actually knows what it means. So, what is it and what is its relationship to Analytics? The key here is information overload: there is simply too much data being collected for most applications, such that it becomes a challenge just to make sense of it.

Thus the treatment of Big Data involves extracting what is relevant to the user, much like the objective of Analytics. However, in the case of Big Data the amount of data is huge, such that the application of Analytics is intended to both display and analyze a relevant portion of that data coherently.

In particular, a given outcome can be predicted with a specific confidence interval (e.g. “there is a 76% likelihood that this unit will fail within 2 weeks”) and remedial action can be prescribed accordingly.  Thus the 3 aspects of Analytics can be applied to the Big Data, in order to recognize, predict and correct problems as they occur.       

Deploying Cloud applications to treat Big Data using Analytics

Typically the above scenarios are addressed via a Web-based Portal or Dashboard display, which allows registered users to monitor their systems and alerts them if there is an unusual situation.  This usually involves a customizable graphic user interface (GUI), which can be divided up into parts that correspond to the specific points of interest for users.

For example, critical data might include:

- Operating parameters of HVAC equipment for an energy management application

- Fuel consumption for a fleet of vehicles or

- Physiological measurements for a medical monitoring system

All of these systems might be accessed from a central server that provides customized Portals or Dashboards for a given set of registered users. This is, by definition, a cloud application and it is increasingly the preferred choice for deploying robust Web applications.

Dealing with government/industry Regulations and Security

After these users log on securely, they should be presented with the specific data that is of interest to them, in a format that makes sense. In other words, they are not expected to waste their time trying to understand superfluous information that is not relevant to their goals.

As a result, the treatment of Big Data via Analytics usually requires considerable subject matter expertise, which may or may not be regulated by government authorities.

This information is typically sensitive, such that measures to deal with security and encryption then become necessary. Furthermore, the presence of large numbers of simultaneous users will require the system to be both scalable and optimized for performance.

Finally, in addition to accessing the information that is available on their personalized Portal or Dashboard via a browser, users will also likely want access to this via a mobile device or tablet instead.

Developing mobile and platform-independent applications

In order to make these applications accessible on either mobile or other platforms, they must be developed with multi-platform use in mind. With a Cloud-based application, the server-side software becomes transparent to the user, who can access it via a browser on a PC or Mac, as well as a tablet or mobile device.

Thus the server can run the development environment (be it Microsoft C# or Open Source Java) and the operating system (such as Linux or Windows) of choice. Hence tools like Knockout GS and Kendo UI can be used to make the applications portable, in order to ensure that each user’s platform is identified and the available screen space is optimized.

Designing the application with a Model View Controller (MVC) paradigm helps ensure that there is a separation of concerns for the development team. In other words, the various developers on the team might focus exclusively on the client side, back-end or middle-ware.

Fine-tuning the user interface

Development of the GUI can be made even more granular, by using the Model-View-View-Model (MVVM) approach, which initially grew out of the Windows Presentation Foundation (WPF) initiative. This approach further breaks the GUI layer down into its own model, view and controller paradigms.

The use of specialized Java Script tools (such as JQuery and Ajax) allows further fine-tuning of the client-side software, thereby diminishing the load on the server and enhancing the response time for end users. For example, JQuery allows “accordion” type screen sections that dynamically select only relevant blocks of information for display on the screen.

Similarly, Ajax can be used to dynamically refresh only specific sections of the display, as required, thereby avoiding the need to annoy users by frequently refreshing the entire page. The result is that the user can experience roughly the same kind of performance on whatever platform he or she is using at any given time.

Aiming for a satisfying user experience

Thus the bottom line is that users must feel that their time is optimized and that they are as comfortable as possible with the application. For example, a surgeon checking a patient’s status while golfing on his day off can access the same information on his smart phone that he’d see on his PC or tablet at work.  

Similarly, a salesperson in the field can use her smart phone to access the same mission-critical information that she’d have at her desk, back in the office. By the same token, the system should be intuitive enough that new users can benefit from it as soon as possible, without the need for a steep learning curve.

That is a critical factor for sales as well, since your sales force will need access to easily understandable and demonstrable prototypes of the system, in order to show it to potential or existing clients. Whatever the circumstances, the use of these technologies and development strategies can result in significantly increased profits, by providing your customers with value-added features that zero in on precisely what they need to know.  

----------------------

[1] From: apmdigest.com/gartner-top-10-strategic-technology-trends-for-2013-big-data-cloud-analytics-and-mobile 

How to analyze a software application for weaknesses and come up with a plan to stabilize it?

Since this is a very wide-ranging question, I will narrow it down to the range of mobile applications that are most relevant to the example chosen here, which is a mobile medical records application. To start with, let’s consider the following scenario:

a)   A tablet-based application with a Windows-based user interface.

This is essentially a Windows application, but which allows the user the convenience of a pen- based interface. Let’s assume that the pen based interface works fine. Then, from a technical perspective, the present analysis can mostly focus on whether the application meets the user’s business requirements and on how well it performs.

Firstly, the performance is irrelevant if the application does not meet the needs of the user! Therefore, here are some questions to ponder:

•     Does the weakness that we are considering stem from an inability to meet these requirements?

•     Do we have clearly documented business and/or functional requirements to refer back to?

•     Was the correlation between requirements and functionality meticulously tracked during analysis, design and development?

•     Do we have documented Use Cases to explain the appropriate role playing for this application?

If we did our jobs properly in the first place, then the answer to all of these questions is “yes” and then we can simply match up the actual functionality to the business requirements. This should allow us to determine where the weakness lies and how to address it.

However, to make this more interesting, let’s assume now that we’re dealing with a legacy system that was ported by the company to a mobile application. In that case, we may be dealing with a system that has no proper documentation to refer to.

Here is where proper quality assurance comes into play! Since we’re assuming that the normal process of rigorous testing was never done on the original app, it becomes incumbent upon our intrepid QA person/people to test the heck out of it, in order to determine where it breaks.

That’s the first step in pinpointing where the app is weak, if we have no specs to refer back to. If we do have specs then the task is much easier, but the principle of using rigorous testing to determine when it breaks still applies.

Likewise, if feedback from the users was not part of the development process (which it would have been if we using an Agile methodology!!!), then it is essential to get that feedback now. In other words, if people are already using the app, then we need to know exactly when/how it does or does not meet their needs.

Again, I must emphasize that this should have happened before the app was released, but failing that, let’s get this input now. This will lead to a set of functional requirements that can then be compared with the test results for our QA staff. Essentially the take home message here is this: a combined effort by the analysts, designers, developers and QA staff results in an application that has no surprises. Ideally this would happen before release, but better late than never!

Once all of this has been done, then we can fix the bugs and/or enhance the product, as needed. At this point we know what it actually does versus what it should do, such that it is now a straightforward process to determine what needs to be done in order to stabilize it (i.e. it’s understood that we’re including bug tracking as a subset of this whole process).

Of course, now we come to the ubiquitous issues of maintenance, such as:

•     How long will it take?

•     How much will it cost?

•     Who will do the work?

•     Who will pay for it/

•     When will it be done?

Sorry to sound like broken record, but once again Agile development comes to the rescue here. After all, fixing or enhancing an app is just a microcosm of the original, overall development process. Thus, the same principles apply:

•     Address any unknown/risky issues first

•     Get constant feedback from the users

•     Ensure effective communication within the dev team, to identify/address problems early

•     Release small, frequent iterations

•     Test constantly!

b)   How corporate strategy directly affects this situation

Another important consideration here is, “What are the relevant apps of our competitors doing?” Thus the above process should also take into account whatever the competition currently does and whether our product is up to snuff.

Indeed, it may even be that the competition has recently added features that make the latest release of our app obsolete! In that case, our app may work just fine and yet still have a significant weakness. Similarly, the technology behind the app must constantly be re-evaluated, with respect to what is currently available “out there”.

The best designed and implemented app in the world may become useless if the technology is

no longer appropriate (e.g. the Yellow Pages). So, this suggests that s/w development should not stand alone: marketing, strategy, product management and other factors need to be considered as well.

The most effective way to ensure that this occurs is to create a system of overlapping competencies, so that all the bases are covered by more than one individual (i.e. like baseball players backing each other up on the field). This requires us to foster an atmosphere of “covering each other’s backs”, as opposed to “looking over each other’s shoulders”. There’s a subtle but critical difference between the two approaches!

Hence, for example, we should have the product manager involved in quality assurance, the development team leader concerned about business management, the chief technology officer wondering about corporate strategy, and so on. Since that appears to be the case already, then the challenge is to propagate this mentality going forward, as we grow.

This leads to two further questions, right off the bat:

•     How do we maintain the sense of family/mutual assistance as we grow from a small to medium to large company?

•     Why hasn’t this company grown more since its inception?

These are tough questions that are best addressed when we meet. However, I will say this here: if we can’t maintain a congenial atmosphere, then the issue of growth becomes moot (unless we want to become just another big, impersonal corporation).

Conversely, that growth may have been inhibited (intentionally or otherwise) by a desire to maintain that atmosphere. In that case, since our products and the related market are both clearly conducive to growth, then the challenge is clear (which is where I come in!).

c)   Addressing the above scenario with apps developed for more specialized devices, such as the

Blackberry, iPhone or iPad

This issue is also beyond the scope of the present discussion and thus best addressed when we meet, but a few quick notes are appropriate here:

•     How do we address potential problems with proprietary software? By the way, this also applies to the use of products from potential partners (e.g. Procura, Qualicode and Motion Computing).

•     What tools are available to diagnose potential hardware problems that can affect our software products? As trusted partners of RIM, do we have complete access to their diagnostic tools? Similarly, what if the initial assumption above was invalid and the pen based interface was actually the problem – what tools does Motion Computing provide for analyzing their pad computers?

•     How relevant is the user’s perspective? Since our products are intended for various user categories (agencies, home care nurses, personal support workers), then the weaknesses in a given app may depend largely on the user’s role.

•     Is language an issue? By developing multilingual products, we expose ourselves to potential glitches that derive from the improper mapping of database objects to the application code. On the other hand, if we did/do this right, then we can just as easily accommodate ten languages as two (i.e. the relational database design should make the mapping for numerous languages as simple as for two). Qualicode is now looking to the European market – are we? What about Asia? 1

•     Why are we not developing apps for Apple products? Unless this is prohibited by one

of our partners (e.g. RIM), then what would be involved in this? Would all of the above apply if we were to develop apps for Apple products as well?

•     What is the meaning of life?

Okay, maybe that last one was just a bit of a stretch … but the answers to these and many more questions will be forthcoming over the course of our upcoming meeting(s). I’ll be that you just can’t wait!

1 Actually, it occurs to me that the 7 languages which I either speak or understand are all European … so, if you want to wait a bit to address the Asian market (until I get a chance to learn Mandarin also), then that’s okay.