Came across a presentation on payment processing by Voca at InfoQ. Just felt that the key design principles used by voca may be of interest to those generally deal with the problems of high volume transaction processing. These principles are:
1. Minimize movement of data: Movement of data across physical and logical layers could result in heavy network traffic and also necessitates a complex transaction and exception management over multiple layers. The idea is to whenever a set of transactions need to be processed for certain validation or transformation, do it within the database instead of moving such set of data to other layers and then bringing it back to the database.
2. Task parallelization: This an area which most of us might have not considered. Using the Work Manager / worker architectural pattern, the tasks can be executed by multiple nodes, which could be separate physical nodes with the ability to add more nodes when demanded.
3. Physically partition data: When the all in one database hits its scalability limits, the option is to partition the database. It would be ideal to envision the possible physical partitioning of the data and implement it right from the beginning.
4. Optimized reads and writes of volatile data: This is one principle, which most of us adhere to by having necessary indexes, managing the fetch sizes, etc.
5. Minimize contention: Contention is certainly an avoidable thing as this will cause the workers wait for release of the resources or data and directly impacting performance. One option is not to wait for the release of resources and instead, look for an alternate source of data / resources. Of course this will require a thought through design of multiple synchronized instances providing the data / resource.
6. Asynchronous decoupling: Usage of a middleware like message queue can certainly help in this area and there by improving the response for the consuming applications.
7. Keep Complex business logic outside database: Considering the limited scalability options of the database, it would be ideal to minimize the work load for the database by shifting it to other tiers where possible.
8. Caching frequently accessed data: There is no point in traversing through multiple physical and logical layers to fetch the same data multiple times. Caching such data in appropriate layers will certainly will leave the network and other resources for other useful tasks.
The referenced presentation is available on InfoQ website at http://www.infoq.com/presentations/qcon-voca-architecture-spring
Here is a presentation on using Disruptor concurrency pattern and achieve a very low latency for high volume transactions.
ReplyDeleteVery nice compilation of the design best practices that will eventually result in a scalable and performing application. To handle high volume, performance and scalability are the two very important attribute that should be taken care of in the design phase. Thanks for writing this up.
ReplyDeleteSanjay
Sanjay,
ReplyDeleteThanks for visiting and commenting on my blog and glad that you liked this post.
Kannan.