What is Agile software development?

A concise but thorough overview of the dominant software development methodology from Scrum to stand-ups, strengths and potential pitfalls

Agile software development is an iterative approach to creating software products based on quickly releasing a minimum viable product (MVP) and then adjusting it and adding features and functionalities in stages based on user behaviour and feedback. The methodology is designed to address the fact it can be difficult to accurately predict the most intuitive user journeys, features and functionalities users need, prefer and desire from software.

As a methodology, agile software development stands in contrast to the once dominant Waterfall approach. When building software to the Waterfall methodology, software development teams create highly detailed specifications and functionality requirements upfront. The software is then built to that blueprint and released as a ‘completed’ product.

The more complex a piece of software, the harder it is to create upfront specifications that don’t miss any details and perfectly anticipate the best options for users. As software products have become more complex over the years the Waterfall approach has fallen out of favour and Agile software development is now the dominant methodology.

An iterative Agile approach helps sidestep the risk of the project sponsor wasting money developing a digital product based on mistaken assumptions of functionalities and features users need and want. Or that is designed in a way users find cumbersome and inconvenient.

Another pillar of the Agile methodology is the promotion of collaborative cross-functional teams. In a non-Agile software development process like the Waterfall approach, there can be limited collaboration between the different skill sets and stages. It’s a much more linear process, like a manufacturing production line.

Waterfall vs Agile in software development infographic

 

A product is conceptualised, designed and detailed specifications created by one group of specialists. It is then passed on to front and back end software developers and designers who build their separate sections which are then put together. That is then passed over to QA and testing teams and, finally, an operations team that deploy it as a live software product.

An Agile software development team is far more integrated with constant back-and-forth collaboration and often regular cross-over between specialists of different disciplines. First the MVP then iterations are planned, designed, built, tested and released quickly in a cyclical process.

The new features, functionalities, changes, fixes and improvements of each iteration rely on constant communication and close collaboration between the specialists at each stage of the development process.

Agile methodology for software and app development

Agile software development follows ‘the scientific method’

As a software development or more general project management methodology, Agile is often compared to the scientific method and it is a pretty good analogy.

The scientific method asks a question and potential answers to the question are put forward. These answers are called hypotheses and are based on all the known information available at the start of the process. A hypothesis is, essentially, an educated guess.

Scientists then try to prove or disprove the hypotheses. That’s done by collecting measurable, empirical evidence gathered from experiments related to each hypothesis. These experiments often take the form of if/then statements and the results either support or contradict the hypothesis.

The Agile frameworks

Agile is a broad umbrella term that encompasses frameworks and practices that represent different nuances of how the methodology is approached. Some prefer to define Agile as a mindset with the ‘methodology’ rooted in frameworks (also sometimes referred to as methodologies) that the software development process follows.

All Agile frameworks are informed by the 2001-published Manifesto for Agile Software Development and follow its 12 Principles but also differ from each other in ways that can make one the preferred approach for a particular organisation or development project.

There are over 50 formally detailed Agile frameworks but a large majority of software development teams and projects use one of the several most established.  These include:

  • Scrum
  • Kanban
  • Extreme Programming (XP)
  • Feature-Driven Development (FDD)
  • Dynamic Systems Development Method (DSDM)
  • Adaptive Software Development (ASD)
  • Rapid Application Development (RAD)
  • Crystal
  • Scaled Agile Framework (SAFe)

Scrum

Scrum, named after a rugby scrum for the metaphor of a whole team pushing together in one direction (the sporting metaphor can be extended to a Scrum team learning through its experiences, positive and negative, to continually improve and organising itself to be more than the sum of its parts), is by far the most commonly used Agile process framework in the context of software development.

Agile software development with Scrum is distinguished from other Agile frameworks and processes by the specific concepts and practices, divided into the three categories of Roles, Artifacts, and Time Boxes. Another key quality of Scrum is the introduction of the roles Scrum Master and Product Owner who organise and manage the rest of the Scrum team including software developers, designers, QA, testing and operations.

Sprints, which are ‘time-boxed’ periods of often around two weeks during which a pre-determined amount of development work is completed, are integral to the Scrum framework. Sprints are Scrum’s mechanism to ensure the Agile principles of “delivering working software frequently”, and “responding to change over following a plan” are adhered to.

Diagram or visual representation of a Scrum sprint

Source: Atlassian Agile Coach

After each Sprint, which ideally results in a new iteration on the software being released into production, an Agile software development team reassesses the needs and priorities of the project. Based on that the next Sprint is planned.

Kanban

Kanban, the name comes from the Japanese word for card, is a framework for ‘just in time’ (JIT) work processes, that dates back to Toyota’s factory floor. In fact, Toyota adapted the approach from a system already being used by supermarkets to make sure their shelves and warehouses were optimally stocked to meet customer needs while minimising carrying excess stock.

When particular components or materials used in the production process ran low in Toyota factories, someone would pass a card, or kanban, onto the warehouse who would prepare a new batch to be taken to the factory. The warehouse would then pass another kanban onto their own supplier when necessary.

The system created efficiencies by making sure the right resources were always available while minimising waste.

In the context of Agile software development, the Kanban system has been adapted as a workflow visualisation system. Kanban cards are digitalised and placed on a Kanban board. Work items being represented visually means team members can quickly see the state of every task at any time, with the board acting as a ‘single source of truth’ for the current state of the software development team’s work.

A Kanban board has three workflow states – To Do, In Progress and Done. Which helps standardise workflow and quickly identify dependencies and blockers that need to be resolved.

The core difference between Kanban and Scrum is that Kanban prioritises a continuous work and delivery flow while Scrum’s Sprints break a development project down into chunks. The emphasis on continuous delivery and continuous integration, CI/CD, makes Kanban particularly popular with DevOps software development teams.

There’s also a hybrid approached dubbed ‘Scrumban’ that attempts to blend the two frameworks by combining Scrum’s Sprints with Kanban’s prioritisation of optimising work in progress limits and cycle times.

Extreme Programming (XP)

Extreme Programming (XP) is not a traditional Agile framework because it places more emphasis on the technical aspects of software development and implementation of specific practices than the management and organisational sides.

An XP software development life cycle has 5 stages:

  • Planning
  • Designing
  • Coding
  • Testing
  • Listening

XP teams also sometimes include the relatively unique role of ‘tracker’, which is often assigned to one of the developers on a part-time basis. The tracker follows metrics like velocity, overtime worked, tests pass rate or anything else the team feels is important to follow progress and identify potential improvements.

XP’s 5 values are:

  • Communication
  • Simplicity
  • Feedback
  • Courage
  • Respect

And its practices are:

  • Sit together
  • Whole team
  • Informative workspace
  • Energised work
  • Pair programming
  • Stories
  • Weekly cycle
  • Quarterly cycle
  • Slack
  • Ten-minute build
  • Continuous Integration (CI)
  • Test-first programming
  • Incremental design

Done Wells, the author of extremeprogramming.org, says XP may be an appropriate Agile software development approach for projects with the following characteristics:

  • Dynamically changing software requirements
  • Risks caused by fixed time projects using new technology
  • Small, co-located extended development team
  • The technology you are using allows for automated unit and functional tests

Feature Driven Development (FDD)

Feature Driven Development (FDD) organises Agile software development around completing features. In the FDD context, features are more like Scrum stories than traditionally defined product features. Mike Dixon gives examples of FDD features on a site for flash video games as:

  • the user needs to be able to play a game on a web page
  • a user needs to be able to play all games on a web page
  • site to have a premium look and feel

Completing the first feature would involve:

“…creating a static html page with some embed code that would allow a user to play a game. The game needed to be in a web facing folder. Once complete and tested, the feature can then be released.”

An FDD project lifecycle follows the five stages of:

  1. Develop an overall model
  2. Build a features list
  3. Plan by feature
  4. Design by feature
  5. Build by feature

FDD is normally considered most suitable for large Agile teams with pre-defined development standards. It’s not considered suited to smaller projects.

Dynamic Systems Development Model (DMDM)

Dynamic Systems Development Model (DMDM) is an evolution of the Rapid Application Development (RAD) framework designed to offer added governance and discipline to an iterative approach to software development.

It is based on the philosophy “any project must be aligned to clearly defined strategic goals and focus upon early delivery of real benefits to the business” and supported by the 8 principles:

  1. Focus on the business need
  2. Deliver on time
  3. Collaborate
  4. Never compromise quality
  5. Build incrementally from firm foundations
  6. Develop iteratively
  7. Communicate continuously and clearly
  8. Demonstrate control

A DSDM Agile software development life cycle consists of 4 stages:

  • Feasibility and business study
  • Functional model / prototype iteration
  • Design and build iteration
  • Implementation

Adaptive Software Development (ASD)

Adaptive Software Development (ASD) is an Agile software development framework designed by the project managers John Highsmith and Sam Bayer in the early 90s. It is a more iterative evolution of the Rapid Application Development Agile framework, designed around 1-month projects broken down into week-long iteration periods (comparable to Scrum sprints).

ASD places a focus on deeply integrated user feedback and involving them in different phases of the development lifecycle. It also seeks to encourage emergence – unplanned new directions and features.

Crystal

Developed for IBM in 1991 by Alistair Cockburn, one of the original Agile ‘influencers, the Crystal Agile software development framework focuses on individuals and their interactions rather than processes and tools. It’s underpinned by the two core beliefs:

  1. Teams can find ways on their own to improve and optimize their workflows
  2. Every project is unique and always changing, which is why that project’s team is best suited to determine how it will tackle the work

Boiled down, the Crystal approach rejects universal development processes and strategies and instead offers guidelines for the collaboration and communication that allows a software development team to decide on its own step-by-by processes and strategies as per their assessment of the project’s unique needs.

One weakness some find in the Crystal framework’s approach is that the fact it de-emphasises the importance of documentation and reporting can reduce organisation-wide transparency and the ability of anyone external to track progress.

Scaled Agile Framework (SAFe)

The Scaled Agile Framework (SAFe) is the dominant approach to scaling Agile software development for large projects that involve multiple Agile teams. The framework has four variations, Essential SAFe, Large Solution SAFe, Portfolio SAFe, and Full SAFe, with each tailored to a different level of scale.

SAFe promotes alignment, collaboration, and delivery across large numbers of Agile teams and draws from agile software development, lean product development, and systems thinking.

SAFe’s values are:

  • Alignment
  • Built-in quality
  • Transparency
  • Program execution
  • Leadership

Its 9 principles are:

  1. Take an economic view
  2. Apply systems thinking
  3. Assume variability and preserve options
  4. Build incrementally with fast, integrated learning cycles
  5. Base milestones on objective evaluation of working systems
  6. Visualize and limit Work in Process (WIP), reduce batch sizes, and manage queue lengths
  7. Apply cadence, synchronize with cross-domain planning
  8. Unlock the intrinsic motivation of knowledge workers
  9. Decentralize decision making

Scaled Agile, Inc., has developed a SAFe implementation roadmap that follows recommended 12 steps:

  1. Reaching the Tipping Point (buy-in for organisational change)
  2. Train Lean-Agile change agents
  3. Train executives, managers and leaders
  4. Create a Lean-Agile centre of excellence
  5. Identify value streams and ARTs
  6. Create the implementation plan
  7. Prepare for ART launch
  8. Train teams and launch the ART
  9. Coach ART execution
  10. Launch more ARTs and value streams
  11. Extend to the portfolio
  12. Accelerate

Other scaled agile frameworks that offer variations on SAFe, include Large-Scale Scrum (LeSS) and Scrum-of-Scrums (SoS).

Advantages of an Agile approach to software development

Agile software development in one of its flavours has not become the dominant methodology in contemporary software development by accident. Software development teams generally prefer to work to an Agile methodology because it generally, implemented correctly, leads to better processes and better software products.

Project sponsors like it for the same reason and because it allows for faster releases, happier, more integrated development teams, and optimises the allocation of expensive resources by reducing waste. A 2018 PwC report on Agile project delivery calculated Agile projects as being 28% more successful against a mix of important metrics.

Nine of the most frequently cited advantages of Agile software development are:

  1. Superior product quality
  2. Customer satisfaction
  3. Better control
  4. Improved project predictability
  5. Reduced risks
  6. Increased flexibility
  7. Continuous improvement
  8. Better team morale
  9. More relevant metrics

At first glance, some of these benefits can seem counterintuitive. For example, if a central pillar of Agile software development is that user stories and features and functions requirements can be changed at any time based on live testing and feedback loops, how can project predictability be improved?

The answer is that adaptability and flexibility at the development level throughout iterations mean the project can be informed by vital new information that wasn’t available at the outset. Better collaboration and improved visibility of what different team members are doing also makes predicting performance and risks, then mitigating for them, easier.

Pitfalls to avoid in Agile software development

Does an Agile software development approach also have weaknesses and potential pitfalls to be wary of? Of course it does. A methodology is only ever as good as its execution. These are three potential dangers to mitigate during an Agile development process.

Scope creep and a loss of focus

Many of the benefits that result from an Agile approach are a direct result of the inherent flexibility and freedom it offers. But the take-it-as-it-comes approach of the next iteration being informed by previous release and feedback loops also means there is inherently never a really clear vision of the final product or outcomes.

The lack of a finite ‘end’ point can see a lack of focus creep into Agile teams, leading to a loss of intensity in work, missed deadlines, and challenges in what to track progress against. Project scope can also start to spiral.

Hard to predict costs, time and resources required at the beginning of large projects

The lack of a formal design phase and the fact Agile is premised on product sponsors and development teams not knowing exactly what the end result will look like and consist of means it is hard to accurately predict final cost, time and other resources that will be needed.

Agile project managers can protect against scope creep and a loss of focus by setting project-specific KPIs to track or by building a product road map. Project sponsors who both want their software development team to adopt the Agile methodology but need to respect business requirements for an approximate budget and deadline might also consider an adapted controlled or fixed price Agile approach.

Difficult to maintain collaboration levels

Agile promotes the empowerment and autonomy of team members through self-organisation and cross-functionality. But for that to remain coordinated and efficient in the absence of linear phase completion requires maintaining high levels of communication and collaboration.

Maintaining the necessary level of collaboration across the duration of a development project, especially a large one, takes effort and commitment. That’s, even more, the case if an Agile team is distributed because some or all of its members are working remotely, as has become increasingly common.

Different Agile frameworks have different tools designed to maintain a high level of collaboration, like the Scrum and Kanban boards.

How do Agile and DevOps work together?

People often refer to either Agile software development teams or DevOps software development teams but it is increasingly often the case that a software development team is both. So how exactly to Agile and DevOps, both of which are defined as project management philosophies and methodologies, interrelate?

By now you should have a good understanding of Agile. DevOps draws a lot from Agile but has a core focus on unifying software development teams and the operations specialists responsible for the smooth running of a software application once deployed.

In DevOps, the development team doesn’t just hand over the software tested in the development environment to the operations team and wash their hands of it. Both teams work as a single unit with same goal of an error and bug-free user experience and responsibility for achieving it.

You can read more about how Agile and DevOps work together but, ultimately, DevOps is inherently an Agile software development approach but Agile software development is not necessarily DevOps.

Featured blog posts