Continous Integration – A blast from the past

Although this didn’t happen a decade ago but still has been a good case for me to learn and realize how Continuous Integration brings value addition to our work.

As I recall it was like this when they were teenagers 😀
Few teams working on different modules of same application, deployed together.
No build process formalized
No builds except for the ones that need major milestone deployments.
No feedback, reports, reviews etc.

So virtually there was no build eco system.

Which meant hideous amounts of communication, tons of trouble shooting, last minute show stoppers, And extermely unhappy end users/stake holders.

So what exactly was going wrong.

There was no Configuration Control i.e. No way to figure out the changes that will be part of the release and no controls to include or exclude them.
No Status accounting i.e. No feedback or status of anything as a whole product.
No management of Environment meant no one knew what hardware/software they were using and sometimes why?
No team work whatsoever meant no guided processes, reviews or tracebility of issues and pitfalls.

So virtually a release day was the doomsday!

It took some time for developers to realize that how important the following were.
* Configuration identification – What code are we working with?
* Configuration control – Controlling the release of a product and its changes.
* Status accounting – Recording and reporting the status of components.
* Review – Ensuring completeness and consistency among components.
* Build management – Managing the process and tools used for builds.
* Process management – Ensuring adherence to the organization’s development process.
* Environment management – Managing the software and hardware that host our system.
* Teamwork – Facilitate team interactions related to the process.
* Defect tracking – Making sure every defect has traceability back to the source

And after all it takes some time but the day to day flow looked like the figure below.

Build Management
Build Management

Importantly
Developers are distributed on different locations in different teams.
There is an SCM team that takes care of the SCM activities i.e. Configuration Identification, Control, Build Management etc.
There is one single repository for all developers.
A process is defined for Product integration and its checked whenever some one checks in.
And hourly build is dont to check the sanity of the system with the unit tests.
A Nightly build is done with all full blown unit tests, coverage reports, Automated tests and installers and upgrades.
More over all of the above is shown on an FTP/HTTP site. FTP for installer/binary/archive pickup and HTTP for detailed view of builds/feedbacks etc.

All of the above meant strength and health of the build eco system. And more trust in the system that it will report failures along the way.

Doing the Locale – Danmark

The following illustrates how to get the Number format working with a danish locale.

  1. import java.text.NumberFormat;
  2. import java.util.Currency;
  3. import java.util.Locale;
  4.  
  5.  
  6. public class TestLocale {
  7.  
  8.  public static void main(String args[]){
  9.  // Create a Locale for Danmark
  10.  Locale DANMARK = new Locale("da","DK");
  11.  
  12.  // get the currency instance for this locale.
  13.  Currency krone = Currency.getInstance(DANMARK);
  14.  
  15.  // Get a Number format for the locale.
  16.  NumberFormat krFormat = NumberFormat.getCurrencyInstance(DANMARK);
  17.  // A symbol for the currency
  18.  String symbol = krFormat.getCurrency().getSymbol();
  19.  // A double amount
  20.  double amount = 10000.25;
  21. // print it out after formatting.
  22.  System.out.println(krFormat.format(amount));
  23.  }
  24. }

That thing about CI – Continous Integration

Challenging Business requirements and the need for software development teams to remain agile and competitive while managing parallel development and releases requires a system which is adaptive to these demands.

Our approach to SCM enables unlimited and adaptable process models, which are ideally suited for parallel, distributed, and agile software development. Using state of the Art technologies from various vendors to automate processes such as branching, merging, build, and release keeps you a click away from software delivery.

The real matter then is the sanity of your application and that’s the challenge we all need to address. We can have cute products right of the shelf to get it up and going but the whole challenge lies in the statement “up and going”. It takes a ton of energy and effort to come up with the right process and the right tool. I have not seen alot of CI projects failing strictly. But I do have seen extra efforts into things that should not be really there. failure would then mean the amounts of efforts spent on doing not so trivial stuff and on the other end if you are spending alot of time doing the major flows then definately something is wrong.

Automation does solve most parts of the problems but things that really need to be taken care of to get it right in the sweet spot are the ones Martin Fowler described in one of his articles.

* Maintain a Single Source Repository
Keep everyone together, at the end of the day we all work together dont we?

* Automate the Build
Automation of the build helps you.

* Make Your Build Self-Testing
Self testing build tells you exactly what you did and what you broke. So I can come up with this statement to my Lead. “If I commit this, it might break alot of stuff”. Well Sir, try running the Unit Test, You will know for sure! But then again honesty on the test cases really doest matter!

* Everyone Commits Every Day
Typically in a Agile environment helps alot, and forces you not to break the build. Better organization I must say.

* Every Commit Should Build the Mainline on an Integration Machine
Sanity checking of your commits. So often we commit without realiazing that we break something else.

* Keep the Build Fast
I usually commit before leaving for home. It helps great deal when you get the feedback within 5 mins. kind of a benchmark but depends.

* Make it Easy for Anyone to Get the Latest Executable
Keep all the build results and one sharable place. We did that on one of the http servers some time back

* Everyone can see what’s happening
Importantly that all results are published for all of us to know.

* Automate Deployment
Deployment automation means that my buddy X on the other side doesnt screw up the server for even a 1 min. Only deploys when build successfull.

Hope this helps.

Resources:

http://martinfowler.com/articles/continuousIntegration.html

How to read a file from the JAR?

Someone just asked me this question today. And I thought might as well put it down for info.

  1.  public TestReadFileFromJar() throws FileNotFoundException, IOException {
  2.         InputStream is = getClass().getResource("txtData/states.properties");
  3.         read(is);
  4. }

In the case above txtData is placed in the jar on the root. Remmember to add the path with the /

Command

By using the command pattern you are seperating the operation from the invoking object. And just because of that it becomes easier to change the command without chagning the caller/s.
This means that you could use Command pattern when you might have the following situation

You want to parameterize objects to perform an action
You want to specify, execute and queue requests at different times.

Just to quickly start you need a command object, An interface will keep it easy going in this case, thus providing you with the option of extending other classes e.g. Swing MenuItem or Button.
Below the execute Method is the one invoked to do something when this command is called or asked to do its stuff.
Where as the getCommandName is assumed as a unique name how ever I am sure we can always come up with a better implementation for uniqueness.

  1. public interface Command {
  2.  
  3.     public void execute();
  4.     public String getCommandName();
  5.  
  6. }

And example implementation of the Command should look as follows
A Command Name, and and execute Method to tell what happens when this command is called.

  1. public class ForwardCmd implements Command {
  2.  
  3.    private String COMMAND_NAME = "Back";
  4.  
  5.    public BackCmd() {
  6.        super();
  7.    }
  8.  
  9.    public String getCommandName() {
  10.        return COMMAND_NAME;
  11.     }
  12.  
  13.     public void execute() {
  14.         System.out.println("Your wish, my command");
  15.     }
  16. }

The command manager is the controller in this case. It registers command objects. the “registerCommand” will simply take a command and store it in a list or something alike. This means you could load it out of a jar file, or an xml or path and just pass the object to the “registerCommand” AS a command offcourse.

the “execute” Command will simply execute the Command passed to it.

And the “getCommand” returns a command by looking up a COMMAND_NAME. So if you provide a name to it through you system it should give you an object of type Command and simple pass it to execute. Again this would be a controller logic and not the client one.

  1. public abstract class AbstractCommandManager {
  2.  
  3.     public abstract void registerCommand(Command command);
  4.     public abstract Collection getAllCommands();
  5.     public abstract void execute(Command command);
  6.     public abstract Command getCommand(String name);
  7. }

Implementing the adapter

Typically when implementing an interface you would have to implement all the methods that exist in that interface. A very good example is the MouseListener in the java Swing. When you need to implement more then one method where as typically you might be catching only one of them. Saying that you would also find a Mouse Adapter provided as well. Some of us use that often. And that is part of the Adapter pattern. It makes life easier for me sometimes.

Adapter a structural pattern will let you adapt to a different environment. The joining between different environment is called Adapter. Thus basically giving others the interface that they expect or vice versa when your program becomes the client.

For example the following class expects that the implementing class should be implementing all three methods.

  1. public interface RecordListener {
  2.  
  3. public void eventPrePerformed(RecordEvent recordEvent);
  4.  
  5. public void eventPerformed(RecordEvent recordEvent);}
  6.  
  7. public void eventPostPerformed(RecordEvent recordEvent);
  8.  
  9. }

So lets say our implementing class is a rude one and only wants to implement one method. What do you do as an API designer. hmmm

Thats where we step in with the Adapter.

  1. public abstract class RecordAdapter implements RecordListener {
  2.  
  3. public void eventPrePerformed(RecordEvent recordEvent) {}
  4. public void eventPerformed(RecordEvent recordEvent) {}
  5. public void eventPostPerformed(RecordEvent recordEvent) {}
  6.  
  7. }
  8.  
  9. public MyAdapterImpl extends RecordAdapter{
  10.  
  11. public void eventPerformed(RecordEvent recordEvent){}
  12.  
  13. }

Now the only thing left to do is use the adapter. And override any method that you might need .

  1. public MyClientClass {
  2.  
  3. public MyClientClass(){
  4.  
  5. this.addRecordListener(new MyAdapterImpl());
  6.  
  7. }
  8.  
  9. }

wasprofile -create -delete

Sometimes you require to do things silently, without any questions asked and “Just Do It” attitude is required.

I often find my self with this problem.

If you want to delete or create a Websphere profile from your command line try the following. (I have tried on RSA only)

# deleteing a profile
wasprofile -delete -profileName MyProfile

You should get the following message on deletion

INSTCONFSUCCESS: Success: The profile no longer exists.

Creating a websphere profile

wasprofile -create -profileName MyProfile -profilePath \
[PROFILE PATH] -templatePath \
[RSA HOME]runtimes\base_v6\profileTemplates\default \
-nodeName [NODE NAME] -cellName [CELL NAME] -hostName [HOSTNAME].

Abstract Factory pattern

Factories have been a key pattern in building applications, its fascinatingly simple, effective and to the point. When starting to learn a design oriented approach to applications or API, I would always recommend a factory pattern as one of the key starting notes of highlight in your design.

So today I am talking about the Abstract Factory pattern. Its not an “abstract” class or object that you call a pattern. But its a Factory of facotries and that is what exactly makes it so much wordingly abstract. Having “abstract” classes is there but just some other side of the coin.

When should I use an Abstract Factory:

  • Independence of how products are created, composed or represented
  • Should be configurable with one of the multiple families or products
  • You need enforcable constraints for the products used as a group
  • You need to reveal only the interfaces of products and not thier implementation as part of a bigger picture.

So lets begin with the fun.

This is how I plan to implement it:
Has A:
Product has a Specification
Factory has a Product
FactoryManager has FactoryConstants
FactoryManager has ComputerFactory

Is A:
BFactory is a ComputerFactory
AFactory is a ComputerFactory

Not shown.
ProductA is a Product
ProductB is a Product

Diagram:

AbstractFactory
AbstractFactory

Creating a simple factory that returns products.

  1. public abstract class ComputerFactory {
  2.  
  3.  public abstract String getName();
  4.  
  5.  public abstract Product[] getProducts();
  6.  
  7.  public abstract Product getProduct(int ProductID);
  8.  
  9. }

Implementation of the ComputerFactory

  1. public class AFactory extends ComputerFactory {
  2.  
  3. public String getName(){
  4. return "A";
  5. }
  6.  
  7. public Product[] getProducts(){
  8. return null;
  9. }
  10.  
  11. public Product getProduct(int productID){
  12. switch(productID){
  13. case 1:
  14. return new ProductA();
  15.  
  16. case 2:
  17. return new ProductB();
  18.  
  19. default:
  20. throw new IllegalArgumentException("Sorry you hit the wrong factory, we closed down in 1600 BC");
  21. }
  22. }
  23. }

A register base for factories. Refer to the main method for use later in this post.

  1. public interface FactoryConstants {
  2.  
  3.  public int A = 1;
  4.  public int B = 2;
  5.  
  6. }

The main Entrant class. the Factory Manager that will give the ComputerFactory resultant. Its assumed to be a Singleton as it registers as a Creator in the system (assumption).

  1. public class FactoryManager{
  2.  
  3.  private static FactoryManager factoryManager = null;
  4.  
  5.  private FactoryManager(){
  6.  
  7.  }
  8.  
  9.  public static FactoryManager getInstance(){
  10.   if(factoryManager != null){
  11.    return factoryManager;
  12.   }
  13.   else return factoryManager = new FactoryManager();
  14.  }
  15.  
  16.  public ComputerFactory getFactory(int factory) throws IllegalArgumentException{
  17.  
  18.   switch(factory){
  19.    case FactoryConstants.A:
  20.    return new IBMFactory();
  21.  
  22.    case FactoryConstants.B:
  23.    return new SUNFactory();
  24.  
  25.    default:
  26.    throw new IllegalArgumentException("Sorry you hit the wrong factory, we closed down in 1600 BC");
  27.   } 
  28.  }
  29. }

A main method to test the AbstractFactory

  1.  public static void main(String args[]){
  2.  
  3.   System.out.println(FactoryManager.getInstance().getFactory(FactoryConstants.A).getName());
  4.   System.out.println(FactoryManager.getInstance().getFactory(FactoryConstants.B).getName());
  5.   System.out.println(FactoryManager.getInstance().getFactory(3).getName());
  6.  
  7.  }

You can find the complete code listing here:
AbstractFactory source

Crocus – CSV Reader

Easy to use ready to go CSV File Reading utility. Read One or Multiple files into a RecordManager, quick access to the file with segmentation into Fields and Records. Merge Multiple CSV files in one. Listener to CSV Files.

Download Here

Organization:
A CSV file is broken up as follows
A CSVField has a group of characters
A CSVRecord has a group of CSVFields
A CSVFile has a group of record

How To Use:

Reading a Single CSVFile into a RecordManager

  1.  // Creating a single file interface
  2. CSVSingleFileInterface fileInterface = new CSVSingleFileInterface("/media/data/dev/workspace/crocus/testData/drupal-sample.csv",CSVConstants.COMMA);
  3. // Calling the read returns a CSVRecordManager i.e. in memory
  4. AbstractCSVRecordManager manager = fileInterface.read();

Reading multiple files into one RecordManager

  1. // Specify a FileSet
  2. AbstractCSVFileSet fileSet = new CSVFileSet();
  3.  
  4. // Add files to the set
  5. fileSet.addFile("testData/drupal-sample.csv");
  6. fileSet.addFile("testData/countries.csv");
  7.  
  8. // Add the fileSet to a Reading Interface
  9. CSVFileSetInterface fileSetInterface = new CSVFileSetInterface(fileSet);
  10.  
  11. //  Reading returns a manager same as in a Single file case.
  12. AbstractCSVRecordManager manager = fileSetInterface.read();

This functionality is not complete but a peak is available.
You can now specify a Listener to Pre, Post and On Add of a record.

Setting up a Listener.

To add you listener simply implement the RecordListener class

  1. // Get a Record Manager
  2. AbstractCSVRecordManager manager = fileInterface.getRecordManager();
  3.  
  4. // Add a listener to the Manager.
  5. manager.addRecordListener(this);
  1. // Implement an event listening method for listening to the RecordEvent.
  2. public void eventPerformed(RecordEvent recordEvent) {
  3. System.out.println(recordEvent.toString());
  4.  
  5. }

for further information refer to the docs.  http://www.shaafshah.com/crocus/

The Build System:

The build script supports the following targets

Build: init, clean, compile, jar, javadoc, tests

Including, Excluding files while creating compile, jar, tests

FILE DETAILS (Paths and description):

PRE RULES:

=========

CROCUS_DEV = the main directory i.e. examples from here onwards this will be the variable used for describing details.

You will need to set CROCUS_DEV env variable inorder to run the build process

Also you would need to set ANT_HOME for usage of ant. I have used Ant 1.6.5 for builds.

$CROCUS_DEV/src Holds the source code i.e. java files

$CROCUS_DEV/build Carries all the build related files

$CROCUS_DEV/build/build.xml Main build script

$CROCUS_DEV/build/include.xml patterns for inclusion of java files at the time of compilation

$CROCUS_DEV/build/exclude.xml patterns for exclusion of java files at the time of compilation

$CROCUS_DEV/build/tests_include.xml patterns for inclusion of test java files at the time of compilation

$CROCUS_DEV/build/tests_exclude.xml patterns for exclusion of test java files at the time of compilation

$CROCUS_DEV/build/tools tools used during build.

$CROCUS_DEV/build/jar_buildfiles you can simply specify a txt file with some wild card matches and name it as yourjar.jar for the build system to recognize that “yourjar.jar” will be the name for this jar that should have the packages in it as specified in this txt(yourjar.jar)

$CROCUS_DEV/build/bin Holds the uniz script and the bat file for developers to run the build script more or less as:

Windows:> %CROCUS_DEV%/build all

Linux:> $CROCUS_DEV/build.sh all

$CROCUS_DEV/build_results Holds all the results of the builds

$CROCUS_DEV/build_results/docs Created java docs

$CROCUS_DEV/build_results/classes Created classes

$CROCUS_DEV/build_results/tests Unit test results

$CROCUS_DEV/build_results/jars the jars created by the system

$CROCUS_DEV/jars/ Is the directory for jars related information

$CROCUS_DEV/jars/manifests Holds the manifests for the $CROCUS_DEV/build/jar_buildfiles.

Conventionally should use: Manifest.jarName

$CROCUS_DEV/jars/original3rdparty Holds any 3rd party vendor jars that might be used for putting in the classpath for the build system

$CROCUS_DEV/testData logically should hold all testData no matter how that might be. currently I have placed a few csv files (tabbed,comma,semicolon).

SVN – some quick commands during merge.

Following are some of the frequently used svn commands during merging and branching. I used to work with tortoise for this but as soon as I learned these, it feels like a more easier space to be in. Although no points taken away from tortoise, it still works pretty good for the gui part, This article is more targeted towards dark screen lovers.

Create a new branch from trunk:
If you want to create a branch from a specific revision of Trunk following command is handy. It does a remote copy. which means the machine you are on does not need a copy of the whole tree.

$ svn copy -r REVISION TRUNK_URL NEW_BRANCH_URL --username USERNAME --password PASSWORD -m MESSAGE

An example

$ svn copy -r 1234 http://shaafshah.com/trunk http://shaafshah.com/branches/MY_BRANCH --username foo --password bar -m "Remote copy"

When was this branch created?
If you want to know the day branch was created.

$ svn log -v --stop-on-copy BRANCH_URL

The last record will show you the day the branch as created.

List all the branches:
If you want to take a listing of branches or a tree

$ svn ls BRANCHES_URL

e.g. http://shaafshah.com/branches

Merge from Branch to Trunk:

Browse to where you have checkedout trunk in the local directory.

$ cd shaafshah.com/trunk

Update trunk to HEAD.

$ svn update
At revision 1234.

Following will merge from branch to trunk but will not commit.

$ svn merge -r BRANCH_REVISION:TRUNK_REVISION BRANCH_URL

Branch_REVISION will be the revision branch was created if this is the first time you are doing the merge.

$ svn merge -r 1233:1234 http://shaafshah.com/branches/my_new_branch

After this you should do an

svn status

to check the status of the files. the files will be marked with following Characters.

‘A’ Added
‘C’ Conflicted
‘D’ Deleted
‘I’ Ignored
‘M’ Modified

If there is any ‘C’ in the status the files will not be committed if you try an svn commit to save the merge to the trunk.

Merge from Trunk to Branch:

To merge from Trunk to branch you would need to browse to the branch checked out in the local direcotry.

$ cd shaafshah.com/branches/mybranch
$ svn update
At revision 1234.

The following command will try merging trunk from revision 1233 i.e. the day branch was created to branch head.

$ svn merge -r 1233:HEAD TrunkURL BRANCH_URL

Hopefully this should help. However you should definitely refer to SVN Book for more detail.

svnbook at http://svnbook.red-bean.com/