In the recent era of web technologies, mostly the SaaS-based applications are developed. The web applications primarily depend on frameworks like Django, Flask, Bottle, etc., especially in the case of Python.
While developing a web application, most developers would have surely come across this concept. Yes, I am talking about threading.
Contents
Pythong Multithreading
Let me start with how threading works with a small real-time example.
I have been working on a project, where the process is scheduled at a certain time range. Let’s say for every 5 minutes.
I was using PycURL to hit the respective process function at the scheduled time. Initially, I was using PycURL for calling the function.
Example:
Process 1: Scheduled at 11:30 PM
Process 2: Scheduled at 11:30 PM
According to PycURL Process 1 will start at 11:30:00 PM and Process 2 will begin at 11:30:02 PM.
As we can see the time difference between the two Processes is 2 seconds, but is scheduled at 11:30 PM. This is fine for the 2 Processes.
Let’s say we have some 100 processes that are scheduled precisely at 11:30 PM.
According to PycURL, the batches will be executed as follows
Process (1, 2, 3 ….. 100): Started at (11:30:00, 11:30:02, 11:30:04, ….. 11:33:20) PM
So the average time difference to start the 100th batch is 3 minutes 20 Seconds.
Problem Statement
I need to start all the Processes exactly at the same time i.e 11:30:00 PM
Solution
And once again, I was confused as to which one to use for starting all the batches at the same time. Parallel execution can be done in two ways: Multiprocessing and Multithreading.
The threading module uses threads, and the multiprocessing module uses processes. The difference is that the threads run in the same memory space, while the processes have a separate memory. This makes it a bit harder to share objects between processes with multiprocessing. Spawning processes is a bit slower than spawning threads.
So, I most certainly would go for Multithreading. Let’s start with a small example of the thread, where I will begin to function 100 times at the same time.
import thread
def my_func(x,ind):
print “Thread “ + str(ind) + “ started”
return
times = range(1, 100)
a_lock = thread.allocate_lock()
with a_lock:
for index, i in enumerate(times):
thread.start_new_thread(my_func, (i,index))
thread.exit_thread()
The above program will execute all the threads at the same time.
In my case, when I used it in my functions, it worked fine, but when I tried to use it in the flask framework, I faced some challenges.
The frameworks will most likely use API calls often to pass data or get data. When I get certain services like Inventory management data, the data is extensive.
Say, for inventory, it takes some 20 API calls to complete the process. During that time, the first API request will work fine, but during another API request for the same service, I faced two types of errors from the flask side.
RuntimeError: Working outside of request context.
This typically means that you have attempted to use the functionality that needed an active HTTP request. Consult the documentation on testing for information about how to avoid this problem.
RuntimeError: Working outside of application context.
This typically means that you have attempted to use the functionality that needed to interface with the current application object in some way.
To solve this, set up an application context with app.app_context().
This is because, when you use it in frameworks, especially in flask there are some contexts, which keep track of the application-level data during a request, CLI command, or other activity.
When we are making requests, especially via threading, the first time execution will run successfully. During the second time request, the app context will be cleared or unavailable.
In that case, you have to hold or recreate the app and request throughout the function execution. Rather than passing the application around to each function, the Application Context is used or called.
The Request Context keeps track of the request-level data during a request. A corresponding application context is pushed when a request context is pushed.
When pushing a request context, you should pass the URL you want to hit to the test_request_context parameter.
with
app.app_context() and app.test_request_context(‘[request_url]‘):
def fuc():
Pass
In the above example, each time the function is called, the app context and request context will be called upon.
Hope this article was useful to you. Please get back to me for any queries.
About DCKAP Integrator:
Improve integration efficiency. Boost customer engagement.
DCKAP Integrator is the leader in B2B & B2C integration, creating a unified integration experience across eCommerce, ERP & CRM applications. The ERP Integration Platform built to solve your day-to-day integration challenges.
Today, more than a handful of businesses use DCKAP Integrator to solve their ERP integration challenges – including MDS HVAC, AQI International, Technico, Howard Supply, Kay Lighting, Cathy’s Concepts and more.
We support Microsoft Dynamics 365, Oracle Netsuite, Distribution One, SAP B1, Epicor Eclipse, Epicor P21, SAP Hybris, Salesforce Commerce Cloud, Magento, Shopify, BigCommerce, Hubspot and many more applications.
