BlockChain

Blockchain is the technology that backs bitcoin and other crypto currencies allowing for a peer to peer financial system. ok so it works for money but what else would you like to be able to make a peer 2 peer system for? blockchain technology can be used and adapted to help solve many problems.

Problem

In a centralised system the server is the trusted third party, so when you send money to another person the server stores the record of how much money you have. We trust that server to let you know how much you have and make sure others don't abuse the system. So we don't need to trust every one that uses the system just the server.

In a peer to peer system we know that any node we connect to could be untrustworthy, but somehow we have to validate what is true or not within a system where multiple nodes could be trying to abuse the system.

Solution

The solution has several parts

• Group updates together into blocks, with each block having a single parent.
• Make every node validate every update (block) based on the systems rules.
• Make it random (ish) which node creates the next block.
• Make it difficult to produce a block so that people cannot rewrite history.

Block
Each set of updates or transactions is grouped together into a block, with each block referencing the block that came before it, This way with the latest block you can trace back the history of any updates.

Making it difficult
To make it difficult to create a block we introduce a problem that will need to be solved before a block can be produced, we make the problem hard to solve and easy to verify, like opening a combination lock without knowing the key, spinning each dial to all possible positions until the lock opens takes much time, but is easy for someone else to confirm that you solved it correctly. To do this we take the data of the block and generate a SHA-1 Hash of it, we then see if it matches some criteria, I.E. it starts with 0000, if it doesn't we increment a number inside of the block and try again, This is essentially what bitcoin miners are doing, solving millions hash calculations.

Making it Random (ish)
Because of the nature of hash calculations some people's potential blocks will solve much faster than other peoples, this means that even if you control much of the processing power of the network, you should still get beat some times.

Handling network Lag
Sometimes different parts of the network can create different blocks at the same time, this can cause there to be 2 valid but differing histories, we consolidate these histories by allowing nodes creating new blocks to work on creating new blocks on top of the block they saw first, after time it is unlikely that the differing blocks will have more blocks piled on at exactly the same time. So we can solve the difference by saying the truth is the block with the longest history, or if they histories are the same length assume it to be the one you saw first temporarily.

Basic Networking
Each nodes connects to other nodes via TCP, and broadcasts anything it would like to add to the network (Completed Blocks, transactions), it also validates and re broadcasts any data that it receives. Thus most nodes will know about a transaction before it is validated. Transactions are often considered valid after a certain number of conformations (block length after transaction added). Because the longer the chain after it the more unlikely a duplicate block will overtake it as the leader.

Recommended Reading

These are the best books I have read so far that I think every software developer should read. Every one of them has really helped me to understand and improve the way I work.

Code Complete (2nd Edition) amazon

Would recommend this book to anyone who is looking at taking a career in software development. Covers almost everything you need to know to a basic level, though a couple of sections are a little dated this book really helps make the transition from just understanding how to code to being a career software developer.

Clean Code amazon

Clean code is a good book about coding standards and thinking about how you lay out your code. While I do not necessarily agree with all the recommendations it is spot on with all the areas that you need to consider. It really helps to make you think about how you lay out your code and take some pride in it.

The art of Unit Testing (2nd Edition) amazon

This is a really good book for people who are looking at starting to do automated developer testing, while it may take a couple of reads mixed with a lot of practice eventually it will all make sense. I would recommend avoiding the first edition of this book as the writer changed his approach on a few key areas that really make a difference when trying to get to terms with unit testing.

The Design of Everyday Things (1st Edition) amazon

The design of everyday things is a great introduction to usability and its underlying concepts. I would recommend getting the first edition as I found the second edition much harder to read due to the extra information in it.

Conceptual Blockbusting amazon

This book is not about programming but I would really recommend it anyway as it looks at the way you solve problems and aims to help you with understanding and improving the process. It is also full of little games and exercises that are fun to do.

Design Patterns amazon

This book is a really good introduction to the concept of design patterns and has most of the classic (although some you probably won't ever use) design patterns listed. While the catalogue part of the book can be hard to read it really helps you by giving you an insight to some useful patterns that experienced developers use.

Refactoring amazon

Refactoring is an awesome book although I find the catalogue part that lists all the refactorings mostly useless when you are working in a modern IDE that has short cuts for a lot of them. It gives a good understanding of what refactoring is and why it is so key to being a software developer.

Soft Skills amazon

This book is essentially a motivational self help book written especially for developers, it is a really good read and full of all sorts of useful information on how to deal with the rest of life while having a software developing career. From making a CV to getting your dream job and managing money this book essentially covers everything that's not normally in software books.

Object Thinking amazon

This defiantly feels like more of an advanced book but really helps you to think about the way you design your object oriented programs. Some of the sections about using polymorphism over conditional logic we're quite a mind opener. Am looking forward to reading this one again soon as I feel this book still has much to offer.

The Mythical Man Month amazon

If any book can be described as a classic in the software development field it is this one, a great book that focuses on managing software projects. While much of the information may seem out of date it has so many points that still ring true in today's software development world.

Git Aliases

Git Aliases

You can add extra git commands in the command line to make your life easier using git alias

if you type git config --global -e you will be able to edit these.
they should go inside the [alias] section (add one if it's not there)

Basic Commands

These commands are just simple aliases for anything that I use a lot, mainly because I have to type less to use them.

s = status

See the status of the repository

c = commit

Commit changes to the local repository

cm = commit -m

Commit with a message

Example: 

git cm "Updated some code stuff"  

aa = add -A

Stages all the changes in the workspace.

co = checkout

Checkout a branch or commit.

Example: 

git co master  

Viewing History

lga = log --graph --oneline --all --decorate

Shows the history for the repository in a nice way.

lgab = log --graph --oneline --decorate

Shows the history for the current branch in a nice way.

Pushing & Pulling

plr = pull --rebase

pull but rebase instead of merge, find this useful when working on a branch that have others working in it to avoid merge commits. Makes the history much cleaner. See the internet for arguments about this.

pushup = "!git push --set-upstream origin \"$(git rev-parse --abbrev-ref HEAD)\""

Push a local branch to the origin remote, does what it says on the tin. Use this lots when working within a feature branch/pull request workflow.

Stash & Reset

Most of these I found on Phil Haacks blog

wip = !git add -A && git commit -m 'WIP'

Work in progress commit, commits anything to the current branch so it can be reverted later. Useful if you need to change what you are working on. You could use git stash but I prefer having a proper commit.

load = reset HEAD~1 --mixed

Used to undo the commit that was done with the wip command and put the changes in the working directory.

wipe = !git add -A && git commit -qm 'WIPE SAVEPOINT' && git reset HEAD~1 --hard

Really like this one, use it instead of "git reset --hard" to clear the current working directory. Much better because it still commits the changes before wiping so they will remain in your local repository until they are garbage collected.

References

http://stackoverflow.com/questions/804115/when-do-you-use-git-rebase-instead-of-git-merge
http://haacked.com/archive/2014/07/28/github-flow-aliases/

Running a Release Retrospective

We just shipped the first release of a project that I have been working on out to customers and it was decided it would be a good time to run a release retrospective with all of the teams involved.

Format of the session

3 hours in the afternoon with roughly 30 people attending.

Introduction

Some senior managers came in and just say a few words to kick off the meeting in a positive way. Sometimes retrospectives can turn quite negative so this helped make the point that the release was a success and we are there to look for improvements in the coming versions.

Discussion of outcomes

Discussion of the expected outcomes of the meeting, this was a list of different items with actions we are going to take and owners for those actions. The items were:

• Recommendations (What we did well)
• Knowledge (What we learnt)
• Mysteries (Things that need investigation)
• WTF's (Things that we can improve on)

Next time I would rename recommendations to something else as they came out as recommended things we can do better rather that things we did well that we recommend we keep doing and pass on.

Energizer

We tried energizer for this meeting as described in parts of a retrospective. After some reading we decided to try the Candy Love Energizer (whats better to get people talking than chocolate?).

This worked well and got everyone being talkative, would really recommend this energizer (think its a bit more interesting as most people haven't done it before).

Story of the project

This was a brief presentation with some key dates and statistics of the project, to help focus people on what we were going to be discussing the following breakout section.

This was shorter than hoped but it worked well to help frame the period we were talking about. 

Retrospective Prime Directive

"Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."

Read out the prime directive and explained that the meeting is supposed to be a positive thing looking at what we can do better and improve on rather than blame.

Breakout

We broke up in to 4 teams each with flip board paper and pens with each team to focus on a different area. Each team should identify Recommendations, Knowledge, Mysteries and WTF's with possible actions.

Quality & QA

Looking at our quality management & testing

Process

This is essentially the story of a story, from customer to shipped.

Planning & Release

Looking at how the project was planned and released.

Work & Environment

How the teams interact and work together, inside and outside of the team. as well as generally at working within the company & skill sharing.

Discussion

Each team presents their findings and we discuss as a whole group the proposed actions.

The discussion was very interesting though some of the topics were very developer focused, it was suggested that we should have a developer only meeting first next time to avoid bringing the other team members to boredom.

Actions & Owners

Review the actions and assign owners.

The hardest part of these things always seems the be actually getting something practical out of as many items as possible. To try to help we assigned owners to each item and I will try to chase the owners to make sure progress is made over the coming weeks.

Release Success

Each person gets a piece of paper and writes a 1-10 on it to say how well they thought it went. This provides an anonymous way to gather information on how well everyone who worked on the release believes it has gone.

While this is probably not the best way to judge the release it was interesting to see the information that came in. In future I would suggest maybe using a more simple system (Well, OK, Poor) or something along those lines.

Conclusion

The session went pretty well most of the feedback was very positive. We identified several points and actions that we will be looking at over the period of the next release.

I would highly recommend playing around with the different subjects to suit the number of people you have joining in and areas you want to cover. Hopefully when we do another release retrospective we can try another set and compare the results.

References

http://www.funretrospectives.com/candy-love/
http://blog.crisp.se/2013/01/22/henrikkniberg/how-to-run-a-big-retrospectives
http://joakimsunden.com/2013/01/running-big-retrospectives-at-spotify/
http://www.retrospectives.com/pages/retroPrimeDirective.html
https://www.thoughtworks.com/insights/blog/7-step-agenda-effective-retrospective

Protocol Buffers & Lists

Introduction
For this example I will be using the following example .proto and looking at serializing large repeated data sets in C#.

.proto

 syntax = "proto2";  
   
 package pcdf;  
   
 message Boiler  
 {  
   required string BoilerId = 1;  
   required int32 TableNumber = 2;  
   required string TableName = 3;  
   required string BoilerData = 4;  
   required string BrandName = 5;  
   required string Qualifier = 6;  
   required string ModelName = 7;  
 }  
   
 message AllBoilers  
 {  
   repeated Boiler Boiler = 1;  
 }  

Serialization Code

 var listBuilder = new AllBoilers.Builder();  
   
 for (var i = 0; i < 7000; i++)  
 {  
   var builder = new Boiler.Builder  
   {  
     BoilerId = "ID",  
     TableNumber = 2,  
     TableName = "Name",  
     BoilerData = "Data",  
     BrandName = "Brand",  
     ModelName = "Model",  
     Qualifier = "Qualifier"  
   };  
   
   listBuilder.AddBoiler(builder);  
 }  
   
 var list = listBuilder.Build();  
 var bytes = list.ToByteArray();  
   
 var fileStream = new FileStream(OUTPUT_FILE, FileMode.Create);  
 fileStream.Write(bytes, 0, bytes.Length);  

For the actual tests real test data was used.

Deserialization Code

 var fileStream = new FileStream(OUTPUT_FILE, FileMode.Open);  
   
 var boilers = new AllBoilers.Builder();  
 boilers.MergeFrom(fileStream);  

Comparison
For a list of over 7k items Json.net serializes about 10ms slower on my machine, not quite what I expected from what is supposed to be a super fast binary serializer. Upon further reading it appears Protocol Buffers are not designed to handle large data sets, but this can be worked around by serializing the items individually.

Fix
To fix this we will serialize each item induvidually but add them all to the same stream with the length of the item added before each item.

Fixed Serialize

 var fileStream = new FileStream(OUTPUT_FILE, FileMode.Create);  
   
 for (var i = 0; i < 7000; i++)  
 {  
   var builder = new Boiler.Builder  
   {  
     BoilerId = "ID",  
     TableNumber = 2,  
     TableName = "Name",  
     BoilerData = "Data",  
     BrandName = "Brand",  
     ModelName = "Model",  
     Qualifier = "Qualifier"  
   };  
   
   var bytes = builder.Build().ToByteArray();  
   var intBytes = BitConverter.GetBytes(bytes.Length);  
   
   fileStream.Write(intBytes, 0, intBytes.Length);  
   fileStream.Write(bytes, 0, bytes.Length);  
 }  

Fixed Deserialize

 var fileStream = new FileStream(OUTPUT_FILE, FileMode.Open);  
   
 var list = new List<Boiler>();  
   
 int size;  
 while ((size = GetSize(fileStream)) != -1)  
 {  
   var bytes = new byte[size];  
   fileStream.Read(bytes, 0, bytes.Length);  
   
   var boiler = new Boiler.Builder();  
   boiler.MergeFrom(bytes);  
   
   list.Add(boiler.Build());  
 }  

This updated code resulted in a significant difference (160ms -> 25ms) between JSON.Net and Protocol Buffers.

References
https://developers.google.com/protocol-buffers/
https://developers.google.com/protocol-buffers/docs/techniques#large-data
https://github.com/google/protobuf

Protocol Buffers in C#

Installation
Protocol Buffers can be installed via Nuget,

 PM> Install-Package Google.ProtocolBuffers  

Or search for Protocol Buffers, there is also a Protocol buffers lite package that is smaller and does not use relection (better for some lighter platforms like mono). Note: an option must be set in the .proto file to compile the proto for use with the lite package.

Tools
As well as installing the protocol buffers library this will also download the tools required to compile .proto files and generate C# files from them. The tools will be located where the nuget package was installed to.

.proto
A .proto file will be required to generate the required files for usage. This example .proto defines a message called TestMessage with 3 properties. The package is used like a namespace and will also be used to generate the namespace of the classes in C#.

 syntax = "proto2";  
   
 package test_proto;  
   
 message TestMessage  
 {  
   required int32 Id = 1;  
   required string Name = 2;  
   optional string Details = 3;  
 }  

For more information on defining a .proto file see the protocol buffer v2 spec.

Protoc
To compile a .proto file locate protoc.exe in the tools dir and run it with the following parameters.

 protoc -oitem test.proto  

where item is the name of the output file and test.proto the name of the .proto file. This outputs a compiled version of the .proto file.

ProtoGen
The next step is to generate the .cs files that will contain the generated code that can be used to create and serialize the data.

 .\ProtoGen.exe item  

This should generate a .cs file containing everything you need to read/write to the protocol buffers.

Serialization

 var builder = new TestMessage.Builder();  
   
 builder.Id = 2;  
 builder.Name = "Test";  
 builder.Details = "Some Detials";  
   
 var message = builder.Build();  
   
 var bytes = message.ToByteArray();  

You must create a builder as messages are immutable (Each message generates and internal Builder class), a message can be turned back in to a builder easily wit the ToBuilder() method.

Deserialization

 var builder = new TestMessage.Builder();  
 builder.MergeFrom(bytes);  
 var message = builder.Build();  

You could just read the data from the builder if you wish.

Other Methods
message.ToJson() & message.ToXml()
These can be used to visualize the message in other formats.

message.WriteTo(Stream stream)
This is used to write the bytes to a stream.

References
https://developers.google.com/protocol-buffers/
https://developers.google.com/protocol-buffers/docs/reference/proto2-spec
https://github.com/google/protobuf

Protocol Buffers

Protocol Buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data (official site).

In short protocol buffers are googles in house binary serialization format, designed to serialize small messages fast and with a small data size.

Where to use them
Protocol buffers are great when you need fast serialization and a small payload (where efficiency/speed is important). They are not so good when data formats are likely to change as this will mean you need to discard and recreate schemas and drop backwards compatibility or have multiple schemas for different versions.

Pros

• Fast
• Small Payload
• Many cross platform implementations (c++, Java, C#, python, JS)
• Generators a provided to generate data access classes

Cons

• Schema Enforced (Need to define .proto file)
• More complicated API than some other serializers

Proto Format
Protocol Buffer messages are first defined in a .proto file using the proto language. This can then be used to generate data access objects for your language of choice.

 message Boiler  
 {  
   required string BoilerId = 1;  
   required int32 TableNumber = 2;  
   required string TableName = 3;  
   required string BoilerData = 4;  
   required string BrandName = 5;  
   required string Qualifier = 6;  
   required string ModelName = 7;  
 }  

Fields are described with options (required, optional, repeated, etc..) followed by Type (int32, int64, string), field name then = and an Id number that must be unique. For more information read the v2 spec.

Proto3
The current version is proto2, but Proto3 is current in alpha and being developed (Source code on GitHub) The update will include a new version of the .proto syntax.

Binary vs Text Serialization

 {  
  "Id": "21",  
  "Name": "Test User",  
  "address": {  
   "streetAddress": "Test Street",  
   "city": "Birmingham"  
  }  
 }  

Some formats serialize to text (JSON, XML, etc..) this can be really useful if you need to quickly view and edit the documents without special tooling. Binary serializations will require a tool that can read the document before you can begin to read and manipulate the data. Binary serializations will normally be smaller and more efficient though.

References
https://developers.google.com/protocol-buffers/
https://github.com/google/protobuf
https://developers.google.com/protocol-buffers/docs/reference/proto2-spec