Sunday, 30 October 2016

Talking at WordCamp Europe 2016 about News Corp Australia's migration to WordPress VIP

At the start of 2016 myself and Juan (a friend and colleague) decided it would be a great idea to present on NewsCorp Australia's migration to WordPress at WordCamp Europe in Vienna. We had been through a huge migration project over the previous 18 month, moving Australia's largest media websites over to an enterprise version of WordPress.

Jump forward 2 months and we found out we were accepted, followed by 2 months of planning and cramming 18 months of project work into a 30 minute presentation. Before you know it we were flying out to Vienna from Sydney Australia for a quick 3 day trip at WordCamp Europe - the largest WordCamp event in the world.

Below is the talk we presented and transcript on how we scaled WordPress to support over 3 million posts across 22 sites, with over 16 million users a day.




Transcript

00:00

18 months ago, News Corp Australia, started its journey migrating all of its websites over to WordPres. Both Juan here and myself were on the software engineering team part of that migration project. We had often heard at the start of this project, WordPress doesn't scale. Well our talk today is to prove them wrong, and to demonstrate that WordPress provides a really good base. It's a really solid platform, and if you implement the correct technical solutions to extend WordPress, it can scale really well for large and high volume traffic websites like ours.

00:48

Historically people may have interpreted News Corp Australia to be a little like this, a 90 year old company, potentially focusing a lot on their newspaper side of the business. But over the past 18 months, a few of the people you can see in this photo plus a lot more in News Corp, migrated 22 of Australia largest sites including news.com.au, theaustralian.com.au & foxsports.com.au over to WordPress VIP. We've imported over 3 millions posts, and are receiving around 16 million users a day.

01:26

Why did we move to WordPress VIP? First of all, what is WordPress VIP? WordPress VIP is a hosting option for WordPress sites that is powered by wordpress.com which is an enterprise level hosting. We needed to escape our old CMS, which was unscalable and very expensive to maintain. WordPress was chosen because it is a very reliable tool, and most of our journalists are already familiar with, making training and up skilling across thousands of journalists much easier. We just needed to find a way to scale WordPress to meet the needs of our business.

02:06

Lastly and possibly the best part, with WordPress VIP we didn't need to worry about this. wordpress.com manages our production infrastructure, so updates to the core get directly deployed, we don't need to worry about updating core, or server patches. Our main focus is developing the best new features for our customers.

At a high level we have developed 45 WordPress plugins and 22 themes. apart from our production infrastructure, we run 58 AWS non production environments, which are a combination of EC2, RDS and SQS, where we try to mimic as much as technically possible the Wordpress.com environments. From a deployment perspective, we run 25 deployments to production each week through our CI/CD platform; and we love our unit tests, we have over 5000 and counting.



03:21

Our journey started in November 2014, when we look back at those 18 months, there are 5 key factors that really challenged us. We wish we had of known about these before we started the project. however, encountering these challenges and discovering their solutions were part of our journey, and actually a great team building exercise. I'm not going to lie though, it was a tough tough journey.

We wanted to share our story for 2 main reasons. First, to demonstrate that WordPress can scale for the enterprise, and secondly, to share the challenges we faced and the solutions we implemented.

4:03

We are going to focus on 5 key areas; site build; front-end; content ingestion: authoring - how we are creating and updating new stories; as well as continuous integration and deployments. We are going to talk about the challenges we faced and the solutions we implemented.

4:24

The first talking point, side build. We really needed the most simplistic way for our WordPress admins to manage and organise the content on our sites. Customizer was a great start to this, it provided a really good base, but we needed to extend it, and this is the functionality we are going to take you through. Working really closely with the XWP team who I can see a lot of in the front here. we added a lot of functionality into customizer, a fair bit of which has been moved into WordPress core. Here is a short video demonstrating just some of the few key features that have been added in or may already now exist in WordPress core.

05:02

All of our sites utilize multiple content areas - that is pretty standard stuff. Within each of our content areas, we have this concept of default widgets, or also known as global widgets, which will render on every single page. We are currently on the business page, and we are looking at the right hand rail, it's just a list of stories. If we now go to the news section in the navigation, you will notice that the widgets we just saw and actually the same widgets that were just rendered. So we can spin up a lot of pages that have very similar structure very quickly. Then to extend that, there is a plugin which we developed which allows you to override the widgets on a specific page and content area. We are on the homepage now, and by checking the 'localise to the current page' checkbox, it allows your to add a completely different set of new widgets for a specific content area for that specific page.

05:54

We call these contextual settings and they are stored as a JSON data structure as a custom post type in WordPress. The next thing you are going to see is our homepage, and that is the next big challenge we had. We have a lot of widgets that keep scrolling, and scrolling, and it's maybe a bit too much content, and that's what caused us a few challenges. By default WordPress as I'm sure a few of us know, stores all of its widget data in sidebars which is then stored in WordPress options. Because we have a huge huge homepage, it meant that the amount of data we were storing in WordPress options was more than 1mb. We leverage Memcache quite extensively, and 1mb is unfortunately the limit in Memcache, so as soon as any key has more than 1mb of data, it's not longer getting cached. This meant that for every single page request, we weren't caching WordPress options, which was a huge performance issue for us.

Another plugin was developed, we called this widgets-plus. Essentially it's a new custom post type and it stores all of the widget data now in its own custom post type in WordPress. This means that we can store everything relevant to just a single widget in isolation, and cache it in isolation, we don't need to worry about caching everything as part of WordPress options. Within our previous CMS it took us hours, sometimes a day to build a relatively simple page. within WordPress, we only showed you a very small part of it, but we can build really complex pages in literally a few minutes. It's a huge efficiency boost for our business.

07:24

After a few months we had a solution for site build, now the front-end architecture was really key for us. We needed to make sure when all of the sites were moving across into WordPress, we weren't going to end up with a mess of spaghetti front-end code.


07:38

I'm sure all of you here like us who are developers (front-end or backend) know that PHP as a templating language, isn't necessarily the best approach. It gets really messy really quickly. We needed a templaitng engine to ensure we decoupled our front-end code from our backend. Twig was our templating engine of choice, and there are 2 plugins that power this implementation. The first one is called vip-twig, and this one is responsible for integrating Twig into WordPress core - it is a very standalone plugin. Our next plugin, template-integration is responsible for exposing all of those core WordPress functions into our twig templates. It basically bridges the gap between WordPress core and the data our templates need to render out posts on the front-end. The good part about this is if we need to implement a new templating language because twig is no longer the laungauge to use, we can keep our template-integration plugin as it's all reusable, and we can simply build a new vip-smarty or vip-ejs plugin to manage that integration for that new templating engine.

08:40

We have now built our sites, and we have a templating engine in place, now we need content. We have been working on content ingestion functionality for around 6 months in parallel, and in May 2015, we were moving into beta testing in a VIP production environment, planning on launching our first site in June 2015. As you can see from our slides, the timeline is no where near the far right of the screen - yes we ran into some problems.

09:14

In our first attempt we ran into 2 major problems. First our posts are authored in a different platform, they are not authored in WordPress, and we had a benchmark of 60 seconds that content needed to be published into the WordPress front-end from the moment it was saved. Secondly we were seeing some rare race conditions in content ingestion, which caused content to be duplicated on our WordPress sites.

As well as the above 2 major problems, when we moved into production we needed to do an initial content import of around 300,000 posts per site. We didn't have MySQL access as our sites are hosted on wordpress.com - we couldn't expect them to give us access. It was a challenge to import that amount of information. Lastly, after that we, would have a constant stream of data updates including news stories, videos, photos coming into our sites at a rate of around 16 updates per second.

10:28

We needed an architecture to support this. Here is a simplified diagram (it is a very simplified diagram) of what we have implemented. It can be read from bottom to top. First our journalist will create a story or update it in one of our editorial tools. This post will then be sent to our API platform which manages all of our content. It will organise it, categorise it, and make it available to other platforms to consume it, WordPress being one of those platforms. On top of that, as soon as a message arrives into the API platform, it will send a notification to our SQS system in WordPress informing it of the new story and to 'start' ingesting.

11:18

With the help of the WordPress VIP team, we developed an ingestor daemon, we call it Turbo, because in the beginning it was built using WordPress crons, but wp-crons were not meant to work in that way. Like I said before, we were doing 16 updates a second and wp-cron were not coping with that, they were not fast enough. We needed to create this multi-threaded daemon. This whole process that you are seeing is what we call 'end to end publishing'. With this architecture in place, we have a single entry point for content imports, whether we are doing 300,000 updates in an initial import, or 16 updates a second. We also have with SQS, reduced the amount of duplicate content, actually it is now at zero, and finally we can ingest content in less than 60 seconds, we are basically ingesting content in less than 10 seconds, as soon as a journalist saves a story, it appears on our site within about 6 seconds.

12:38

Another challenge that we faced was the amount of fields and data we had that represented a 'NewsStory', or a 'WordPress post in our business. This is stored in our APIs, this is the object we get. As you can see there is a trimmed down version of how our JSON object looks, it's massive, it's a really big object. At the beginning we thought lets split this into pieces, save everything into post meta fields, and build them up again to send writes back to our APIs. However, we thought, why re-invent the wheel, if we are already using this JSON object across all our other platforms, let's just save it in our WordPress database, and consume it from there, and read it from there. Due to this we decided to store this JSON object in the post_content field in the posts table. With this architecture, we needed to change how our templates work, and how our plugins work with this field. We are no longer reading it as text field, but as a JSON object.

Saving the JSON object here allowed us to have revisions of an entire post object. Also when rendering a post on the page, we can do it with a single database call. We finally kept a simple rule, we would only use post_meta for searchable parameters.

All this gave us a huge performance benefits by more than halving the amount of database calls.

14:19

As you can probably guess, June came and past and we didn't launch our sites. But they stayed in beta testing for a few months. Our internal development team, the XWP team, and WordPress VIP team, worked for around 2 months to implement solutions to all the problems we just discussed; and one night in September, a very very late night, we launched our first site on WordPress VIP. It was a night filled with anxiety, way too much pizza, and then a sigh of relief when it switched on and everything just worked.

By now we have implemented a simplistic site build approach, our front-end is clean, and we have content quickly ingesting into our WordPress sites.

15:21

Now we needed to use WordPress for what it is meant to be used for; for editing and creating content. We developed these authoring screens within WordPress that expose our data JSON structure that defines our News story content so people can edit it in the most simplistic way. As you can see we leveraged meta boxes quite extensively, and used CSS and Javascript to provide the look and feel we were after. We tried not to deviate from WordPress core at all so any updates that come through to WordPress.com did not affect us.

15:58

Here is another image of one of our authoring screens to demonstrate what we are doing for image management within posts. We are leveraging off the media element in WordPress and allowing authors to edit/crop images within the UI.

16:15

Our category management is also a little different, our stories have to have in the permalink, the URL of the specific category they belong to. So we needed to add in a primary category (or section as we call it), that they have to select when creating a story. Then you can put that story in multiple categories if needed.

16:48

Within the first week of February in 2016, we launched our last 4 sites into wordpress.com in just one week. And everything just rolled out very very smoothly. Then gradually over the next 4 months, until pretty much last week, all that authoring functionality that Juan just showed you, rolled out into a production environment.

17:08

So we needed to maintain this complex WordPress platform that we had setup. We also needed to ensure that at all costs it was impossible for any cowboy coders to rise up and try and take control of our codebase.

For this it was really critical that we had a solid continuous integration and deployment platform setup. We needed to make sure that a developer can get his code or her code from their local environment all the way into production in the most stable and risk free way.



17:31

For us it comes down to feature based branching and deployments. Every feature for sits within it's own feature branch, branched off master in GIT. It's always developed in isolation, it's always tested in isolation, it's always deployed in isolation. Our team chose to never bundle up more than one feature and deploy it at once, as for us this added too many risks and too many dependencies.

Our team then utilizes pull requests to control any code merges into master. This is where code reviews happen, which is one of our favourite pastimes. Every single line of code is code reviewed by another developer. No single line of code reaches production without being code reviewed by at least 1, 2 or sometimes 3 other developers in the team.

18:14

We then utilize Atlassians Bamboo continuous integration platform to manage all of our automation. This is where on every code commit we run PHPCS to make sure all of our coding standards are being met, we run PHPUnit to make sure all of our unit tests are passing and there are no regression issues we are aware of. Once all of those tests pass we package up the application, running any front-end build tasks in gulp or grunt, we compile our TWIG templates, then we deploy that out to our 50+ non-production environments.

This is when testing happens, either manual regression testing, or we have a suite of selenium based automation tests using the robot framework. Once all manual and automated tests have passed we deploy this feature through our continuous integration platform all the way into the WordPress VIP environment. We celebrate with a few beers, and then we start that process all over again for the next feature.

The best part is, this isn't the process we started with 18 months ago. This evolved over countless retrospectives, team huddles, and big issues with our previous processes, to what it is today.

19:48

With a solid continuous integration and deployment platform setup, the only other challenge we faced was with WordPress VIP you cannot control the exact time your code is deployed to production, and if your feature has changes in 3, 4 or 5 different code repositories, you can't guarantee the order the code in those repositories are always deployed in.

19:32

We needed a way to be able to turn features on and off in a production environment. Welcome to the feature toggle,  This was developed by one of our developers called Roman back in Australia, and it is an implementation of the feature toggle technique by Martin Fowler. It abstracts all of the complexities around feature toggling. So with just a few lines of code you can setup a new feature, you can give it a nice descriptive name, and in the last parameter (the boolean) you can decide if it is active or deactivated by default.

20:00

Then all of our themes and plugins have access to check if this feature is enabled or not. If it is enabled you can execute a certain chunk of code. Lastly this plugin exposes an admin page in WordPress, that lists all of the features we currently have available to us, it's de-active and activated state, and this is where you can toggle a certain feature on or off. This feature toggle has allowed us to, for our really big features, to roll them out to all of our sites from a deployment perspective, and then gradually toggle them on one site at a time so we can stagger the regression testing required.

20:32

So we have come to the end, and what have we really learnt? First of all, as Juan pointed out, migration projects are hard hard work. No matter matter how much planning you do, you are always going to run into unknowns. For us it was really critical for us to get our code into a beta version on the WordPress VIP infrastructure, so we could test our end to end solutions in their environment, rather than our non-production setup.

To all those people that said to us that WordPress isn't going to scale for what we are doing. Well if you just download WordPress core, install it on a single server an try and support 16 million uses, then well no it isn't going to work. But if you implement the correct solution architecture, with things like memcache, elastic search, asynchronous content ingestion, it scales really well for sites like ours.



21:21

Lastly, us partnering with two really awesome teams. With XWP who really put a lot of effort in at the start of our project, their expertise in WordPress was absolutely priceless for us. They were very pivotal in our project, especially in the first 12 months, and a lot of us have become really good mates.

Secondly, with the WordPress VIP team, they just kept going and going just like the energizer bunny, they never stopped no matter what challenges and issues we threw their way.

21:58

At at very high level that was our teams journey to WordPress over the last 18 months. If anyone has any questions I believe we have some time now otherwise please come find us throughout WordCamp in Vienna if you’re chasing any more details on our talk, or just want to have a chat.





Thursday, 4 August 2016

Experimenting with mob programming at NewsCorp Australia


Mob programming has been gathering more and more momentum over the last 6-12 months so we decided to experiment with it in some of our software development teams at NewsCorp Australia. So what exactly is Mob Programming?

”The basic concept of mob programming is simple: the entire team works as a team together on one task at the time. That is: one team – one (active) keyboard – one screen (projector of course). It’s just like doing full-team pair programming.”


Why mob programming?

How often do specific developers in a team only understand certain aspects of the code? And while code reviews are a great collaboration technique, sometimes you need more. Mob programming is a great technique to solve technical problems together as a team, allowing everyone's input to be voiced. As a positive side affect it can also help to encourage pair programming within a development team.

So how did mob programming go in NewsCorp?

Like anything you rarely find success by forcing a particular practice onto a team, it needs to adopted by a team naturally if they see it really adds value. We started by simply running a few exercises with a few of our front-end and back-end development teams to experiment on how mob programming could work within our business.

Front-end development team

 

Our session had 9 developers (rather large for a single session), all taking part in a JavaScript FizzBuzz challenge. Each developer was driving for a single 3 minute interval (we only had time to go around once).

Opening remarks started with, "ahh Fizzbuzz, this should be easy", and "I'm not sure how I feel about people watching me code".

Round 1: & 2:

Mood: Cautious and careful
It started off fast as the first person to drive took the laptop and started coding.. until... the team quickly pointed out, they have to listen to the navigators. This is when everything slowed down and everyone started discussing, "do we do unit tests first", "can we re-read the instructions in the readme", and "this is harder than you think when we all have to do it together". By the end of the first round, the team had created the start of the "FizzBuzz" method.

By the end of the second round the basic FizzBuzz method was complete.

Round 3:

Mood:  Focused and more opinionated
Seeing the team started with the core functionality, at the start of round 3, questions were being raised about starting unit tests, and how they were going to test this functionality. By the end of round 3, we had an initial unit test that tested the basic 'FizzBuzz' was outputting correctly.

Round 4-5:

Mood: Positive and excited
Round 4 and 5 saw a wider set of unit tests being created to test the validity of Fizz, Buzz and FizzBuzz outputs.

Round 6:

Mood: Calm
There was some refactoring of the FizzBuzz application, as well as some cleanup to the unit tests to improve coverage.

Round 7-8:

Mood: Frantic and fun
By the last 2 rounds the team focused on outputting all of the numbers from 1-100 in the console. Challenge complete!

The teams observations:

The team started off slow in giving directions, well, in the way the driver understood the instructions being given. It took a few rounds for the team members to be talking the same language. After this, all someone had to say was, "created a unit test", "copy that function and past it there", "rename this to that".

As expected, there were the more confident developers navigating more than others at the start, but when it was their turn they immediately became the listeners, which gave an opportunity for other developers to voice up.

The mood was also really positive. At the start people were saying, "oh now everyone sees how I code in real time", but by the end everyone realized their peers were quite open and not judgemental.

Back-end development team:

This session had 5 developers, all taking part in the same FizzBuzz challenge (but in PHP). As before, each developer was driving for a single 3 minute interval, however we went around 2 rotations.

Opening remarks started with, "ok, lets see how this goes"

Round 1, 2 & 3:

Mood: Energetic
Team was keen to get started quickly and smashed through most of the requirements in about 2-3 rounds.

Round 4, 5 & 6:

Mood: Strategic and careful
These rounds saw the team work closely on unit testing to get confidence in the core code that was developed. A lot of discussion around correct architecture also took place, when the team decided what and how to refactor.

Round 7, 8 & 9:

Mood: Agreement, disagreement and focus
Within the last 3 rounds, the team rapidly implemented refactoring to allow better unit testing of the implementation of FizzBuzz.

Observations:

Withing 3 rounds the application was done. It was a quick MVP to get the basics in place. This development team had been working together for over 12 months so they knew how to work with each other very well.

The team took the approach of fleshing out an MVP, then adding unit tests, then refactoring the application to improve scalability. Why did they not do unit testing first? This is a team that has thousands of unit tests around their current code base, and there is a culture that code never reaches production without unit tests. To them, getting a quick MVP with the understanding that unit tests will be there for release is already well accepted.

What the team did uncover, was using sessions like this to flesh out complex features together could really speed up the overall delivery of a future feature. Currently code reviews are heavily used in this team, and this is when potential issues in design decisions are picked up. But having a 1-2 hour session to mob out the basic architecture of a new feature could really reduce feedback in code review process.

What's next? Will we keep mob programming?

This is up for each team to decide... I believe we all agree, sessions like this can be great to get an entire team over a single concept or feature, or to architect a new application with everyones input right from the start.

It was definitely a fun exercise that we will keep experimenting with.


Monday, 14 March 2016

Making code reviews part of your team culture


Code reviews are like unit testing, as developers, we really want to do them, but not every team is able to introduce them successfully. This is a post around a presentation I recently gave at What Do You Know Sydney about how to make code reviews part of your development team culture.

I want to share how code reviews in one of the teams NewsCorp are no longer a manual chore, but are a part of the team culture and are treated in the same way as scoping, development and testing.

I've worked with amazing development leads over my career who dedicate so many hours each and every night code reviewing and providing great feedback on the code developers in the team have developed. Although this works and adds huge value in ensuring the highest level of code, it is not a scalable solution.

What is a code review?

“Using your experience as a developer to review and assume joint ownership of that code”

What do code reviews involve?

  • Checking out the code locally and ensuring it runs as expected.
  • Making sure the code complies to the teams syntax standards.
  • Running the unit tests (and ensuring there are unit tests).
  • Making sure the code architecture & design patterns being used are the right ones.
  • Being 100% certain there are no security vulnerabilities in the code.
  • ...Every team will have it's own interpretation of what a code review entails...

Why do code reviews fail within some teams?

  • It's an ad-hoc process.
  • Developers don't understand what a code review should involve.
  • There is not enough time allocated to code reviews.
  • When only the leads or senior developers are doing the code reviews.
  • Most importantly, when there are no surrounding processes in place to encourage code reviews.

How do you make code reviews part of your development team culture?


Make it part of your task estimation

  • Unit testing is usually around 20% of the development time (on average), we have found that code reviews are around 5% of the development time. If a feature takes a 5 day week to develop, usually half a day is enough to code review this newly developed feature. 
  • Remember to allow some time to implement development feedback from the code review.


Utilize Pull Requests (and enforce approvals before merging)

In their simplest form, pull requests are a mechanism for a developer to notify team members that they have completed a feature. In the open source community this is normally how new changes are submitted to core contributors to review and implement into the core open source platform. Within your team ensuring that at least one developer must code review and approve a Pull Request before it is merged is a great step to encouraging code reviews within a team.

Use a great tool like Stash for code reviews

  • Stash is a great tool from Atlassian that makes code reviews simple.
  • You can configure stash to enforce an approver for every code review before it can be merged. 
  • You can actually take it a step further and not allow developers to merge code (only allow testers to merge Pull Requests, this allows the testing team to control the flow of code that goes into UAT and production environments). 
  • Stash has an awesome UI to make comments on code reviews easy and to even allow adding tasks to Pull Requests that have to be completed before merging.

Add it to your Agile board

  • Don't hide the code review process - keep it out in the open and display it on your Agile board.
  • This encourages the wider business to accept code reviews as part of the day-to-day.
  • If a tasks fails code review it moves back into development.


Use code reviews as a knowledge sharing tool

This is the single greatest value code reviews have given the team. I often hear in development teams that this developer knows this, and that developer knows that. When your team embraces code reviews, you obviously still have people experienced more in different areas of the code, but at least 2 developers understand the same area of the code. This is because 1 developer has coded it, and at least 1 other developer has reviewed it - they have checked it out locally, ran the unit tests, understood the code and gave their tick of approval for it to be released.

How can you introduce code reviews in your team?

  • Don't ask permission just do it!
    As developers it is our responsibility to own building stable, secure and scalable code. If we need to spend time unit testing and code reviewing our code, then that is our responsibility to do so (and fight for it).
  • Use a great tool to encourage code reviews like Stash.
  • Use Pull Requests within your branching strategy and enforce approvals.
  • Don’t have exemptions - it's all or nothing!!
  • Have a reward each sprint or month for the person who does the most amount of code reviews
  • Have a token for code review immunity if you find some developers are getting too many code reviews.
    We use a Jedi robe, if you are the one wearing the Jedi robe (of which there is only 1), then no person can assign you a code review until you take off the robe.
  • Most importantly, be an honest code reviewer.
    Don't just approve a Pull Request, every constructive piece of feedback you add, is making your whole team stronger and better developers.

The team I am referring to in this presentation has code reviewed  over 172,000 lines of code in the last 12 months which in my books is an impressive statistic.


Tuesday, 17 February 2015

How to efficiently build an MVP for your tech startup

It's surprising how often I meet entrepreneurs and start-ups who have been building their tech startup for more than a year (sometimes up to 2 and half years) and they haven't launched yet. When working on a startup you often want to make sure it's perfect before you launch so you can show your customers how amazing your product is. But what happens if they don't like what you have built? How can you not waste years building a product that your customers might not even want?

Words like 'Lean Engineering', 'MVP' & 'Agile development' have become mainstream within the tech startup industry in the last few years. Without getting too technical (but following these practices) here are some tips to stay on track, stay focused and get your MVP out to your customers FAST (within 2-3 months)!

What is an MVP?

MVP stands for "Minimal Viable Product". It refers to what is the simplest product you can design, build & release to your customers with minimum time and cost investment. Think about what will give you the best ROI or validation that what your building is what your customers "actually" want?

MVP is sometimes also referred to as a 'Minimum Feature Set' by some people. Be careful though, an MVP is not a broken featureless site, it is a carefully constructed 'prototype' that is a subset of your large product goal that you can build and release very quickly to "validate" your idea. The image below demonstrates really well how the MVP process should work.


Your goal

Your goal is to build the smallest, useable subset of your product that can be used to test against your 'real customers' to determine if you are building a product they 'actually' want to use (and more importantly, pay for!).

What you want to DO (to deliver fast)

  • Scope and define your MVP requirements (what features are required to launch your MVP)?
  • Stick to your requirements.
  • Define & manage your priorities (on a day-to-day basis with your team).
  • Create 2 TODO/backlog lists:
    • "List 1" is your MVP TODO list (these are must have features for launch).
    • "List 2" is your "After MVP" launch TODO list (these are all the cool features you come up with that aren't critical for launch and can be done after your MVP is released).
  • Define your timeframe.
    • 2-3 months is your ideal timeframe to complete design, development, testing & release.
    • Try to really stick to a 3 month maximum timeframe for your MVP.
  • Collaborate together as much as possible (share an office space).
    • If you collaborate together you will get more done together.
  • Have morning standup meetings.
    • Standup meetings are part of 'Agile project management' and are quick informal catchups where everyone defines what they did yesterday and what they are doing today. They are around 5-10 minutes, everybody must standup, and it usually occurs at the start of the day.
  • Always have a TODO list every day about what you and your team are going to get done
    • Have stop limits that will stop you spending too much time on unimportant tasks.
  • Stick to your requirements.
  • Don't add new features unless they really block your initial MVP launch.
  • If you really need to add a new feature (or someone in the team is pushing for a new feature):
    1. Ask your team, is this feature mission critical for launch?
      • If YES, take out another feature that is less important.
      • If NO, put it in the technical backlog for after launch.
  • Don't rewrite your MVP on new technology stacks unless you physically cannot launch your MVP without doing so.
  • Be doers, not talkers/strategists. Building an MVP requires you to get your hands dirty.
  • Stick to your requirements.
    • I cannot stress this one enough.
    • This is the number 1 reason every single startup I have consulted to has struggled to launch their MVP in a timely manner.

What you DON'T want to do (stop getting distracted)

  • Don't keep adding more features to build into your MVP - Keep It Simple.
  • Don't go and build the world (eg: your entire product - focus on the prototype).
  • Don't go and hire a team of 30 developers overseas and think it will get built faster (unless you have a team of 3-4 technical project managers to manage them).
    • Normally finding 1 or 2 developers in your local country will get the job done faster and better.
  • Don't change priorities every day (changing priorities constantly wastes development time and reduces your team efficiencies).

Checklist to get your MVP out FAST!

Based on the startups I have been involved in, below is a check-list of all the things you should consider while planning & building your MVP and then releasing your MVP to your customers. There are approximate time estimates against some tasks to provide you a basic guideline.

Planning & Setup

  • Complete a learn canvas before anything else (~3 hours)
    • This planning tool helps you determine if you have a product worth investing time in.
    • Pollenizer have a great example on their tools page.
  • Determine the team you will need (at a minimum you will most likely require)
    • 1 designer
    • 1 developer (full stack)
    • 1 marketer / salesperson
  • Determine the roles you will need within your team (people can own multiple roles)
    • Design
    • Development
    • Dev ops (servers/web hosting)
    • Product owner
    • QA/Testing
    • Marketing
    • User acquisition
    • Project management
    • Company evangelist
    • "The glue" (the person that brings everything and everyone together).
  • Recruit or source your team (if you haven't got them already).
  • Find an office space (co-working space is a good start) (~1 day).
  • Decide on a company name (that also has a domain name available) (~1 day).
  • Define a long term vision/strategy for your startup - just a sentence or 2 (~2 hours).
    • Print this out and put it on the wall/door, wherever everyone will see it.
  • Define your MVP goal - what you want to achieve for MVP launch (~2 hours).
    • Print this out and put it on the wall/door, wherever everyone will see it.
    • This will keep your team constantly focused on the end goal.
  • Setup google apps/email for all your team (~4 hours).
    • Create a generic email (eg: team@yourcompany.com) that you will use for all your subscriptions, billing etc.. don't use personal accounts (it makes it complicated when people move on in the future).
  • Setup a task tracking or project management tool (~4 hours).
    • Asana is free and simple collaborative tool.
    • If you are after something more enterprise, check out JIRA from Atlassian.
  • Similar to above, setup a collaboration tool (Google Docs or Confluence) for documentation so you don't loose any documentation throughout your early startup stages.
  • Setup an Agile Standup Board 
    • Backlog, TODO, In process, In Testing, Done
  • Don't build the world
    • Find and define that one feature that makes you different from your competition and focus on that.
    • If you're a dating site, what makes you different? Build that feature only.
    • If you're a social network, what gives you an edge to acquire users? Build that feature only.
    • If you're an ecommerce site, what will make your customers purchase off you instead of your competitors?
  • Fake features you are not sure about or cannot build in time:
    • eg: Need "identification checks' on your website?
      • It may take 3-5 days to implement with an API to provide this automatically.
      • For an MVP maybe it's a file upload tool and your team manually does ID checks in the beginning (this may only take a few hours to build). This doesn't scale but remember you're building a prototype.

Design & Development

  • Develop your wireframes for your website/app (~2 days).
  • Define your company brand and colours (~1 day).
  • Develop a company logo (~1 day)
  • Develop your designs and user experience flows for your website/app (~3 days).
  • Research & choose your technology stack and framework (~1 day).
  • Purchase your domain name.
  • Purchase and setup your website/app hosting.
    • AWS provides a great infrastructure to do this and allows you to scale up as required with little effort (provided you architect you're application correctly).
  • Purchase and setup your DNS hosting
  • Define and scope your MVP development build list (~2 days).
  • Prioritise your development tasks in order of what needs to be accomplished first
    • This is a collaboration between your software engineers and your product owner.
    • Break your scope into user stories and tasks.
  • Setup a private GitHub or private BitBucket account for version control.
    • If your developer doesn't know what version control is, find a new developer.
  • Build your MVP (~25 days).
    • Break this into 2 parts.
      • Part 1 = Build the foundation that can be deployed to your test servers and be used as a feedback/review channel for the rest of your team (~15 days).
      • Part 2 = Iterate and build on top of this foundation (from feedback from team) to complete your MVP (~10 days).
  • Ensure your development, testing & production environments are setup (~3 days).
    • Ensure they are all identical.
  • Setup automated deployment (continuous delivery).
  • Setup Google Analytics or another analytics platform
    • Do not launch without this.

Marketing & Acquisition

  • Prepare your marketing plan from day one (~2 days)!
  • Define who your customer is (~1 day).
    • If you don't know who your customer is, figure it out before you building your MVP.
  • Define your customer acquisition strategy (~2 days)
    • How are you going to build your database of customers.
    • Customers are very unlikely to just 'come' to your website - you need a strategy.
  • Talk to everyone about your startup while you are building it.
    • Don't go stealth, don't lock yourself away and think everyone will love it when you launch
      • THIS doesn't happen!
    • Go to meetups, meet people & get feedback from early adopters.
  • Allocate your time into sections.
    • A rough allocation could be:
      • 10% general business setup.
      • 10% wireframes/designs/logos.
      • 5% content creation.
      • 40% software development.
      • 10% QA & testing.
      • 25% marketing / reporting.
  • Setup your social media accounts (~1 day).
  • Setup a landing page to gather signups & social media followers (~2 day).
  • Setup a MailChimp account as a base platform for your subscribers (~4 hours).
    • MailChimp is super easy to use and it's easy to export your subscribers at a later date if you want to migrate to a different platform.
  • Create/practice your elevator pitch (~1 day and then ongoing).
    • Know how to pitch your startup to everyone you meet (but not in an annoying way).
  • Develop marketing strategy plan for launch (~2 days).
    • What newsletters are you going to send?
    • What SEM will you implement?
    • What is your marketing budget?
    • What is your social activity for launch?
    • What giveaways/competitions will you run?
    • What press publications are going out for launch?
  • Setup a customer support tool to install on your website/app.
    • ZenDesk is a great, really cheap tool.
  • Select 10 potential customers for your product that you can use for early feedback (~1 day).
  • Get out there and talk to your customers about your MVP and get feedback to determine if it is worth investing more time and money into your startup idea.

Testing

  • Build a test plan for your MVP (~3 days).
    • This can be a manual test plan in the beginning if you don't have time to build and setup automated testing.
  • Test your MVP on testing servers when it is at a stable stage (~2 days).
  • Do a spell check on your website before launch (~1 day).
  • Release your MVP to production (and do a soft launch) (~2 days).
    • A soft launch is a launch that you invite friends and family to before you release to the public (usually 2-3 days later).

Culture

  • Work hard, if you are serious about your startup, put in a solid 14-16 hours a day in the beginning to get your MVP up and running.
  • Have breaks or dedicate a day off to not think about your startup.
  • Go out to team lunches/drinks at least once a week.

Remember...

Your MVP does not need to be perfect, this is version 1 - "a prototype". This is to test if your customer wants your product so you can justify continuing your startup and not wasting away your life on a product that no one wants.

Suggestions / Feedback?

Have something I have missed that you do when building a startup? Make a comment below and I'll add it to the post :-)

Tuesday, 9 December 2014

How to run stand-ups efficiently

Stand-ups are designed to be a short, fast and efficient meeting each morning to keep projects on track and to expose and provide visibility on blockers to the team, the SCRUM master and the product owner.

Although this sounds relatively straight forward, I've seen and been involved in too many stand-ups that get distracted when its not effectively managed by the SCRUM master. Below are some areas to focus on when running and trying to improve your daily stand-ups.


Each person talks about 3 things only

Each team member has the permission to talk about 3 things only. 
  • What they did yesterday
  • What they are doing today
  • Any blockers they are facing

They do not need to go into technical implementation details, or how they solved a really hard problem. Most people can communicate the 3 points above in less than 60 seconds. Each person needs to focus on communicating to the team the highlights and not the technical details. If this doesn't happen, the SCRUM master needs to intervene, make a note of the discussion subject and take it offline.

Start the stand-up at the same place and same time every morning

Consistency is so important when it comes to stand-ups. Find a time in the day that your entire team is available and a place that is close or next to the task board.
  • If needed, you can get the team to pick the time of the stand-up so that it encourages ownership from the team to make it to the stand-up on time each morning.

Make sure everyone stands up!

Enough said - it will make your stand-ups more efficient.

Communicate with the team, not the task board

Stand-ups are about communicating progress and blockers to the team, not to the SCRUM master and not to the task board.

SCRUM master to record blockers and ensure their resolution

When a person raises a blocker, or a problem, it needs to be recorded, mostly just by the SCRUM master so they can follow up to find out if they need any help resolving, and also more importantly, so that it doesn't get forgotten and that it gets resolved.

Take technical discussions offline

If any person in the team starts going into technical details (or starts talking about their weekend), they need to be stopped by the SCRUM master and told to take the conversation offline after the stand-up. The SCRUM master can take a quick note about the subject, and grab the relevant people after the stand-up to discuss in more detail.

Team members should know what they have to say before the stand-up

Team members shouldn't spend the first 60 seconds of their update trying to remember what they did yesterday, and what they are going to do today. Everyone should prepare before hand and if they need to, come to the meeting with a list of things to talk about.

A team size of 5-8 people is ideal - don't make it larger than 12 people

12 people in a standup is large, but if everyone can stick to 60 second updates you can get it done in the 15 minutes. Essentially you want to make sure that everyone who attends the standup is providing benefit to the other people in the stand-up. If their updates are not relevant or required, think about if they need to be involved in the stand-up. 

No phones or tablets at the stand-up

People participating in the stand-up should be focused and always paying attention - there is no need for people to be on their phone checking emails throughout stand-up.

Last team member to arrive starts first

This is a good practice to help get your team to arrive promptly to stand-ups. If they are the last person to arrive in the morning, they need to go first at the stand-up (as the entire team has had to wait for them). Don't start the stand-up until everyone is present.

Randomise the order of who speaks at the stand-up

You want to keep things interesting and everyone paying attention. Some days you can go left to right, other days right to left. You can even use a 'talking token', we've previously used a football and once you have finished your update you would throw it to the next person of your choosing. People often look for the person that wasn't paying attention to throw it to - it's a great way to keep everyone paying attention.

Use a heavy ball as the 'talking token' if required

An approach to stop people from talking too much and getting into too much detail is to use a heavy object (like a medicine ball) as the 'talking token'. People usually don't want to be holding it for too long so they will tend to be more efficient in their update. Don't make it so heavy that you cause your team back damage though.

Stand close together

Huddle together as a team, there is no need to have a metre of space between each person. The SCRUM master should ensure people don't have to yell to be heard.

The team should run the meeting (not the manager)

Stand-ups aren't about providing updates to the manager (although it is a useful way for managers to stay up to date), it is about the team communicating and providing visibility about user stories to the team. The SCRUM master is there to keep the stand-ups under control and on track.

Lastly, remember to have fun at your stand-ups!

Friday, 5 December 2014

What makes you a senior software engineer?

Are you a senior software engineer?

Software engineers are a sought after resource at the moment. You will sometimes see job adverts stating someone with 4 years+ experience can satisfy the requirements of a senior software engineer. But regardless of experience, what really makes someone a senior software engineer?

1. Accountability and ownership of their code

Ownership of a project is one of the biggest steps a senior software engineer makes. No longer are they a junior engineer that can rely on their coding issues being picked up by senior engineers, now they are the last line of defence for stable, secure applications that are delivered on-time. They are 100% responsible for ensuring that the their own code and their teams code is well tested, scalable and secure.

2. They can solve complex technical problems (any problem)

Senior software engineers have the expertise to look at any problem and find a solution (or undertake the required research to find a solution themselves). They sometimes work with their team members to find a solution, but they never come back to their team lead, or manager without a solution or variation of a solution. I believe this is one of the most distinguishable traits of a senior software engineer.

3. They develop reusable code

Senior software engineers understand that the code they develop may be used through the command line, from a controller in an MVC framework, or even from within a unit test. They develop their code to be used in any environment without duplication. This essentially means building loosely coupled code within the platforms they work in.

Service Orientated Architecture (SOA) is a really good methodology that promotes decoupling of code and reduces code duplication. Also have a read of coding to interfaces to help achieve code loosely coupled code.

4. They code for the future

Senior software engineers approach coding a solution to a problem by looking into how the component they are developing may need to be used in the future or extended in the future. They write their code so it is easily extensible and doesn't require a full re-write to add additional functionality at a later date. This is easier said than done, but after working on a variety of platforms, your past experiences in software development really help you excel in this area.

5. They unit test their code

Whether you have allocated time or not, a senior software engineer should and will unit test their code. They see the benefits of automated testing and will write unit tests to cover off at least the core libraries of their codebase.

Some helpful links on unit testing

6. They can estimate work effort and gather requirements

Work effort estimation is a skill that takes many years to master. It is one skill to be able to accurately estimate work effort for your own tasks, but being able to effectively estimate work effort for your team members is a really valuable skill. Although in Agile development environments the entire team is involved in work effort estimation, senior software engineers should be able to identify when work effort estimates are too low or too high and voice their concern to correct the situation.

7. They mentor junior developers

It is the responsibility of the senior engineer to always look for continuous improvement within the team. That can be exposing the team to more efficient technologies, helping them write more secure code or ensuring the code is well written and unit tested. This mentorship from senior engineers to other software engineers in the team really helps move the team towards greatness.

8. They have a solid understanding of programming methodologies (not just a language).

Senior software engineers have either studied or been exposed to different programming methodologies. It doesn't matter which language they are coding in, software methodologies cross over the boundary of coding languages. These best practices may enhance scalability of an e-commerce website, or provide secure APIs for mobile applications. It is important to understand that senior software engineers can take their learnings from multiple languages and frameworks and apply them to their current projects. They also find it very easy to learn new languages as they already understand the theory behind programming.

9. They're always looking for areas to automate

Senior software engineers are some of the most experienced technical people within a team. They should have a good understanding of how to automate aspects of their teams work. This can include:

  • Partial automation of code reviews (eg: using GIT hooks to run automated tests of coding standards using tools like PHPCS or JSLint).
  • Unit testing code to ensure its security and stability.
  • Implementing continuous integration and deployment services - Checkout Bamboo by Atlassian

10. Complies with and promotes coding standards

Senior software engineers understand the importance of consistency of code within applications. They are responsible for ensuring coding standards are defined within their team, and that everyone within the team is following them. They should also actively review peers code and coding standards to ensure they are up to date.

11. They can explain problems and solutions to non-technical people

A senior software engineer needs to be able to communicate to non-technical people. Although I have met some great senior engineers that maybe don't excel in people skills, they are still able to communicate through the use of diagrams, presentations or user stories. Communication is a crucial skill in progressing past a senior software engineer role to a team lead role.

12. They are always up to date with latest technology

This is actually a hard thing to consistently maintain as there is so much technology out there that is constantly evolving. However, it is crucial that a senior software engineer does their best to stay updated (at least with technologies in their space). You don't always get to experiment with different technologies within the workplace, but side projects you work on outside of work are an amazing opportunity to explore a new framework or technology. Senior software engineers also take this one step further and look for ways to implement their learnings within their current organisation, as well as up-skilling their team members on what they have learnt.

If you want to read and learn about how to become better a software engineer, check out this great post by Artur Ejsmont.

So if you can cover off most (or all) of the above, you've got a pretty impressive chance of landing that senior software engineer role you have been wanting to apply for.

Good Luck!

Wednesday, 26 November 2014

Ideas on Implementing Agile Development


Do you need Agile Methodologies?

The first question to ask yourself is, "do you need to change your current project management processes?"

Don't change something that is already working efficiently just because Agile development is what everyone is doing. Implement Agile (or any project management process for that matter) as a response to solve a problem that you and your team are facing. If your companies backlog or task tracker is feeling something like the image above, maybe keep reading...

At a high level, possible reasons to implement Agile development methodologies could be:

  • To improve team efficiency.
  • To reduce developer distraction and frustration.
  • To improve team focus and operational efficiency.
  • Improve team moral.
  • Provide better visibility of priorities to your team and wider business.
  • Encourage team ownership and accountability for features they develop, test and release.

Get your team's buy in

The number 1 most important thing to do before you implement an Agile development environment is to get your teams support. Without your teams buy in, you are destined for failure. Demonstrate to your team the benefits of Agile methodologies:

  • Run some informal training sessions with your team.
  • Demonstrate how it can solve problems your team is facing
    • Reduces changing priorities.
    • Can reduce stress within the team.
    • Can deliver a more focused development environment.

It is also just as important to get the wider business to support the changing processes. They need to understand how it will affect them, but more so the benefits they will gain as part of this implementation.

  • Reduction on changing business priorities and more focused efforts.
  • ROI drives priorities which drives development effort.
  • Encourages forward planning.
  • Less important features will be de-prioritised and most likely not built.
  • There are shorter more manageable timeframes that are consistently met by the development team.

Start small but aim big

Agile is a massive suite of project management methodologies that can be implemented in so many different ways. It's not advisable (or possible) to implement every single aspect of Agile development in one go - you are really setting yourself up for failure if you do this.

Agile is about creating the most efficient ways to run projects for your team - and remember every team is completely different. What works in one team may not work for your team. So it is crucial that you start with the basics and constantly iterate and improve the processes to create the greatest efficiency - remove what's not working, and add new processes that do work.

Some advice a very experienced Agile coach told me was, "If your offline process for running projects don't work, then how can you expect your online processes to work?".

What he was driving at, is that you really need to get your Agile processes working offline first before you start implementing any online technologies or project management tools. A gradual roll-out plan I have successfully implemented is loosely documented below:

  1. Spend a few weeks training up your team on Agile methodologies (what it is, how it will help).
  2. Document all of the projects/tasks in a single backlog for your business.
  3. Prioritise all of these projects/tasks with your executive or management team (all department heads need to be on the same page).
  4. Work with your team to estimate the work effort for each priority.
  5. Get the most important projects/tasks and setup a basic Agile task board (Swim lanes can include "TODO",  "In Progress", "Done"). Do this on a physical wall or whiteboard.
  6. Start with a 2 week sprint (try really hard to not change priorities).
  7. Run morning stand-ups each day (no more than 15 minutes a day).
  8. Have a single row up the top of the board that is 'unplanned tasks'. This will demonstrate distractions your team is facing that is limiting them from delivering on the sprint.
  9. Run a retrospective at the end of the 2 week sprint (last day, Friday).
  10. Run sprint planning on the next Monday.

Start with a physical task board, create user stories on cards and sub-tasks on post-it notes. After a few sprints, once you have adapted the processes so that they are working, you can move some of these processes to an online project management suite. Atlassian has some great project management tools for managing Agile teams including JIRA and Confluence.


Continuously improve

Agile isn't about putting in place a process and leaving it. It's about constantly improving the processes every single sprint, week or feature release. Perhaps the most important part of the Agile methodologies are running retrospectives frequently and determining what has worked and what needs improving in the next sprint.


Retrospectives

Retrospectives should include everyone that was part of the sprint (developers, designers, QA, product managers). It should be a constructive and quick meeting at the end of each sprint (usually 30 minutes is enough). If you are running the retrospectives you need to ensure the team focuses on constructive feedback and are not negative in their responses.

Get the team to identify:
  • What went well that the team should continue to do.
  • What needs to be improved next sprint.
  • Each improvement recommendation should come a potential solution to the issue (this encourages your team members to always be thinking about solutions, not just problems). 
As a team you can then vote on which areas should be improved within the next sprint. Assign action points and owners to ensure improvement actually occurs (create tasks for your next sprint if you really need to allocate time).

Summary and learnings about Agile

  • It should create visibility for your team and wider business about priorities to focus on.
  • Allows your team to focus on the priorities at hand and reduce changing priorities.
  • Define priorities based off ROI, operations efficiency or other company goals.
  • Run sprint planning throughout the sprint, not just at the start of the next - this way you are constantly planning.
  • Get your team to estimate task work effort, don't estimate as a manager.
  • Assign ownership for each user story so your team develops ownership and accountability for the work they are delivering.
  • Agile should create an environment that your team can flourish in.
  • Allow your team to release quality features, often! Always be delivering.
  • Always improve - your Agile development processes are never finalised, you should always be looking for and promoting improvement.
  • Always run retrospectives at the end of each sprint.