REST Service Connection Pooling

Nice article about http client connection pool on micro service architecture

http://www.baeldung.com/httpclient-connection-management

http://blog.nigelsim.org/2011/06/17/debugging-connection-pool-leak-in-apache-http-client/

Agile Teams in 2018

How can you create a team that is highly motivated, able to  prioritize on its own and deliver the most important user value in a constant pace indefinitely? 

The answer is - Practice AGILE.   

I have been part of so many successful Agile teams over the past decade,  learned and practiced many invaluable lessons throughout my career. Most recently I was part of an elaborative training with a highly effective team that practices these principles religiously.

These are the some of the most critical aspects to create a successful Agile team from my experience.

Unified Backlog
Keeping the single backlog of all activities is the number one priority. This includes business stories, tech items, even including all project mgmt, environment action items. These should be defined well so anyone in the team is able to work on it.

This reinforces the priority to the entire team. The transparency will enhance the trust in the process. The other aspect is to bring visibility to all non-business activities that need to take place to achieve an item in front of business folks.

Self-organizing teams
The team should be empowered to decide on the priority, methods and means to achieve the priority. As much as possible, external interferences should be reduced if could not be compeltely eliminated.

Team assessment
Anyone will obviously put forward his/her focus and efforts based on what is valued in the environment, on what basis performance is appreciated in the team. Unless we start valuing the collective achievements of the team, it will not translate into teams performance.

Individuals in the team should be able to trust each other. The common objectives should be repeatedly communicated to the team by the management. Instead of individual performance reviews, the peer reviews and overall team achievements should be considered for valuations.

This experience not only enhanced my trust in Agile but also gave me practical
handlers to follow Agile Manifesto Principles as much as possible even in a regulated bureaucratic environments.

ATEA Regional Conference - 2018

Over the memorial day long weekend, I attended a community based regional Entrepreneur Convention.

Great honor to meet some of the great minds and more over it turned out to be an wonderful opportunity to listen to their success stories directly from them.

It was a jam packed 12+ hours event with back to back talks of almost 15+ speakers. However, they did an amazing job of delivering their core message short and sweet due to time constraint.

The blog aims to bring the some of excerpts from the event.
-------------------------------------------------------------------------------------------------------------------------

The most important thing to do is take your time to contemplate daily. Success will follow. 
Assume your responsibility as custodian of the planet. This will bring best out of you.

Chandrika Tandon 

The intersection point of 1) Things you are good at 2) Things you love to do 3) Things for which you are paid most is the sweet spot for your success. 

Ram V Iyer. Founder Business Thinking Institute. 


Do not afraid of fear. Fear is part of the process. It is okay to live with it. 

Suuchi Ramesh. Founder Suuchi Inc

No longer companies with huge capital are driving the trends but the companies that create interface for supply & demand, manage the risk and trust are driving the treands and hence the economy. World is moving towards The Shared Economy. 

Arun Sundararajan. Prof @ NYU

Security Token Offering (STO) are the new way of fund raising. 

Atul Kunwar, CTO Tech Mahindra


Do not start a franchise before talking to both winners and losers who attempted the franchise in your market area.

Parthiban Jayaram


Personalized Medicine is the future.

Vasantha Kumar Ramasamy, Founder, CEO Care Centra


To recruit a great team, meet the candidate few times in different sittings. Anyone can sell you for the first 1 hour but only the smart ones can keep you interested even after 4-5 meetings.

Ashwin Bharath CIO Revature


To succeed in venture capital, Invest in someone who has "unfair advantage"

MV Subramanian. Co-Founder Future Focus. Angel Investor

Attention (no multi task), Generation (self directed), Emotion  and Spacing (Repeat) are the pillars of adult learning. 

Gopinathan C.P - Co-Founder 361 Degree Minds.


Disclaimer : As I don't have an habit of taking notes during the talks (I hate it like taking continuous video on the concert), I try to recollect from my memory and present here in my own words. These are my own interpretations and in no ways fame or defame the speakers.

HTTP : 413 Entity Too Large

Keep your headers (including cookies) short!

Sometime back, on a typical Monday morning, I faced a production issue where one of the user could not able to access our site.

The webserver was rejecting all requests with 413.

After analysis, we found out that we were using dynamic headers for passing list of feature togglers (to indicate what features are available for the users) as part of http headers. Turned out few additional features were turned on over the weekend and this user is kinda of super user to avail most of the features.

This resulted in header size being larger than the Apache allowd limit of 8kb.

As a quickfix, we increased the Apache header size limit to 16kb thus allowing requests with larger header to pass through. For permanent solution, we decided to create dedicated REST API to refer to the feature flags as needed instead of passing them through as part of headers.

Is it a limitation of HTTP? 

No. HTTP specification does not limit length of headers at all.However most web servers do limit size of headers they accept. Depending on web-server and their settings these limits vary from 4KB to 64KB (total for all headers). For example in Apache default limit is 8KB.Server will return 413 Entity Too Large error if headers size exceeds that limit.

What are the default limits for commonly used web servers? 

Apache 2.0, 2.2: 8K
nginx: 4K - 8K
IIS: varies by version, 8K - 16K
Tomcat: varies by version, 8K - 48K

Why web servers need to pose this limit? 

Uncapped HTTP header size keeps the server exposed to attacks and can bring down its capacity to serve organic traffic.

What we learned? 

Do not pass anything and everything as part of headers especially if you know the size is dynamic and can potentially increase over time.

Review the headers and the max combinations and have of sense of max length as for most servers, this limit applies to the sum of the request line and ALL header fields (so keep your cookies short).

Courtesy: https://stackoverflow.com/questions/654921/how-big-can-a-user-agent-string-get

Please share your thoughts.

Manifesto for rich minds!

We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win…

John F. Kennedy, September 1962

Database Batch Update - Pattern

Introduction 

Last week I had an opportunity to program a SQL script which will update certain data to fix a data issue. The volume of the impacted data made it an interesting problem to approach.

Being working on high visible, zero-downtime environment, I do not want to make this fix an yet another DB update.

Instead, I embarked on to create a resusable, configurable, repeatble, test intensive, high-performing package so that it canbe  referred (if not directly re-used) for any such issues in the future.

Design 

The approach can be summarized in following steps

1. Load the record indentifiers of an impacted data set into a temporary table. For example, if you need to update an employee table, you then load the impacted employee_id into the temporary table assuming employee_id is the record identifier (primary key) of the employee table.

2. Create SQL Block (or stored procedure), to loop through the temporary table records via cursor and process the actual updates as required.

3. Execute the Block or Stored Procedure (created in above step) in parallel with multiple threads

4. Periodic commit the data so that if any exception occurs in the middle, we dont end up rolling back everything.

5. Devise extensive Pre and Post validation scripts for testing.

Next few sections elaborates these items.

Temporary Tables 

By preloading the impacted data to temporary tables, we can effectively achieve following things

1. We can precisely work with exact set of data that will be impacted. No need to keep doing expensive query on the original tables. For example, if we need to work on 1 million records on the 10 million data table, the temporary table will just have 1mm data which will be substantially cheaper to execute than on original table.

2. Add status column to track the processing as well as to add re-run capabilities

3. Add a THREAD_ID column through MOD(rownum,THREAD_COUNT) to logical partition the data. This can enable parallel execution whereas each thread can just work the records specific to the THREAD_ID.

4. Distributed execution - Once we get temporary data loaded, we can slice and dice the actual update based on the temp table data and redistribute the actual update through all available nodes.

5. Pre and Post validations becomes much easier to execute as we have capture the status column on the temp table. We can also add additional pre-processing (before update) and post-processing (post update) data on the temp table for additional validations

6. There is however one caveat on stage load. If the data is very dynamic in nature (>1000s of update in a minute or so), we may be working with stale data instead of latest upto data. This risk can be mitigated by re-checking impacted values of specific record before actual update.

Export / Import 

Many prominent DBs allow export of select query into csv file as well as import the data from csv file into table. This provide two main benefits.

1. Export and Import processes are independent thus reducing resource contention.
2. Export can happen on READ ONLY replica - If the select query is complex and costly, this will be very handy as it will not impact other systems.

We need to make sure Character Sets (encoding) are same between the export and import so that we can potentially corrupt the textual data in the transit.

Parallel Processing (Multi Threading)

Single threaded processing does not make sense for any update with reasonable volume. We will not only take advantage of modern multi core processors but will also limit the impact on Online DBs as everyone is trying with 99.99% availability.

To do parallel update, we basically need to write a script and that spins out multiple threads with below steps

1. Open Connection
2. Select What data set of work on - Apparently with THREAD_ID isolation, each thread can work on independent set of data.
3. Execute the actual update (Either through Anonymous Block or Stored Procedure)
4. Update the status in temporary table/batch processing table.
5. Close Connection

Wait for all the threads completion and exit the process. Opening individual connections for each thread if critical here.

In some DBs (DB2), passing input parameters to anonymous SQL blocks is not possible, we need to use Stored Procedure in those cases. Stored Procedure also make the code reusable. We can also make a standard skeleton Stored procedure which can take actual SQL command as input parameter and then through EXECUTE IMMEDIATE, it can run the dynamic SQL.

Testing : Pre and Post Validations Scripts 

Last but not the least, like any other programs, testing is lot more essential in batch updates. Especially, the batch updates may take long hours to complete, we should be able to monitor the progress, kill and re-run as required at any point in time.

I would recommend write these scripts in unit test script fashion with possible assertions and estimated outputs.

For example, Lets say, we track the batch updates through status column on the temporary table, we can write SQLs to track the count and decipher the progress based on that.

SELECT
COUNT(*) as TOTAL_COMPLETED,
COUNT(*)*100/TOTAL_RECORDS PCT_COMPLETED,
FROM
   TEMP_TABLE_STATUS_TRACKER
 WHERE
   STATUS = 'COMPLETED'


It is also essential to perform a volume testing in a prod-like environment before implementation.

Implementation Plan 

Success of any project execution highly depends to availablity of detailed implementation plan. Each step should be written by targetting the person (DBA or DevOps or Production Support) who will execute the steps. The steps should also include how to execute the pre and post validations and if required, how to monitor the progress and all required backout/rollback plans.

Its critically important, to have the steps peer reviewed and possibly executed as it is by potential implementor in lower environments.

Conclusion

Albeit it seemed to be "just another update" on the table, I learned quite a lot on many intricasies of the system, especially

1. Handling huge volumes updates.
2. Faster Processing with multi threaded program.
3. Coding for failures (Thorough Exception and Error Handling).
4. Well defined monitoring and validations.
5. Data Model and System Domain Knowledge.
6. Multiple rounds of Reviews with Peers, SMEs and DBAs. Be Humble to learn.
7. "What" it takes to execute the plan in production in a highly regulated environment.

Above all, I hope to achieve with this blog, how to write a design pattern in pure English without delving into too many technical specifics. Hahaha!!!

Netflix Zuul

In the age of micro services, housekeeping work like logging, monitoring, security etc becomes debatable item on the responsibility.

• Do we create a separate micro service to do that? 
• Do we create a API gate way (parent application) to do that? 
• Even if we create API gate way application, how do we scale 

The answer to these questions comes from micro service pioneer - Netflix. The answer is Zuul 

https://github.com/Netflix/zuul 

Its an open source and its gaining lot of tracking from the open source world. 

Java is moving Javascript way?

Starting with version 5, it seems clearly enough, Java is slowing moving away from core philosophy of Object Oriented Programing towards dynamic functional paradigm.  

Lamda functions is a bold first step!

The proposal to include Java Enhancement Process 286: Local-Variable Type Inference in Java 10 is yet another example. 

Ironically though Java Script looking to improve its type safety with Type Script. 

I wonder, in near future we may be see 

• The core philosophies of all programming languages will converge to provide clean, universal APIs and syntax structure. 
• JVM will embed NodeJs environment 
• Javascript can become JVM compatible language  

Share your thoughts! 

Courtesy : InfoQ on Java10

MyRecycleGuide

Proud to announce the kick-start of my new social initiative MyRecycleGuide.com. The upcoming mobile app will provide all recycling related details based on your location in your fingertips.

Find more at MyRecycleGuide