python imports

Publié le 2 mars 2022 par davidhxxx

Definitions

Module: a file containing Python definitions and statements.
The file name is the module name with the suffix .py appended.
Within a module, the module name (as a string) is available as the value of the global variable __name__.

Import a module( that is all functions and variables definitions)

it is not always the best way to import a function or a variable because:
– bloat dependency: it imports all definitions of the module while we need only some of them.
– less handy: we need to prefix the function or variable by the module imported

Import module with top level module bound(really to avoid because we need to prefix the whole path of the top level module for each function or variable we use in the code) :
Example: 

import files.path_examples
#...
files.path_examples.function_used_only_for_import_testing()

import a module with last level module bound
It is a better way to import a specific module. Because the client code only need to prefix the members by the identifier chosen in the import declaration and not the whole path.
Example: 

import files.path_examples as pe
#...
pe.function_used_only_for_import_testing()

import specific function or variable

import a specific function of a module: 

from files.path_examples import load_a_file_from_current_working_directory
#...
load_a_file_from_current_working_directory('')

import a variable to read it:
beware: this way to import a variable doesn’t allow to see its modifications at runtime.
The variable has a final value at the time it is imported.
To allow that we need to follow the way below (import a variable to modify it). 

from files.file_reader import TAB_LENGTH
#...
def import_a_constant_to_read_it():
    # with this import we do a copy of the value variable, we don't refer to the variable declared in the imported module
    print(f'TAB_LENGTH={TAB_LENGTH}')

import a variable to modify it: 

from files import file_reader # import for modifying
from files.file_reader import TAB_LENGTH # import for reading
#...
def import_a_constant_to_modify_it():
    file_reader.TAB_LENGTH = 8
    print(f'TAB_LENGTH={TAB_LENGTH}')
    print(f'file_reader.TAB_LENGTH={file_reader.TAB_LENGTH}')

Alias an imported function: 

from files.path_examples import load_a_file_from_current_working_directory as load_file
#...
def alias_an_import():
    load_file('')

Debug path problems

PYTHONPATH
the search path for module files.
The format is the same as the shell PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows).
Non-existent directories are silently ignored.

sys.path allows to read or modify the PYTHONPATH value

For example when I run a main() python file located in python-sequence\app and I want to display the PYTHONPATH valueI have this one:

['C:\\programmation\\workspace-python\\python-sequence\\app',
 'C:\\programmation\\workspace-python\\python-sequence',
 'C:\\Python39\\python39.zip',
 'C:\\Python39\\DLLs',
 'C:\\Python39\\lib',
 'C:\\Python39',
 'C:\\programmation\\workspace-python\\python-sequence\\venv',
 'C:\\programmation\\workspace-python\\python-sequence\\venv\\lib\\site-packages']

Lazy versus eager import

Overview

In Python by default imports are eager. It means as soon as we import a Python/module file, this one is loaded.
In the very most of cases, this behavior is nice. But in some specific cases where imports are long to be performed and where we need to make the application ready as soon as possible, using eager imports may be a solution.
How to achieve it?
By importing the module not at the beginning of the Python file user but inside a function or even inside a condition inside a function.

Examples

Suppose you have a Python application taking as argument a string meaning the process (hello or bye)to perform and the username target of the process.
The hello process requires to import a long processing Python file while the bye process requires to import a lightweight Python file.
By using the default import way, in the two cases, the long processing python file load will be performed, even if it is only required for the hello process. 

import sys
from datetime import datetime
from typing import List
 
print(f'{datetime.now()}: before import')
from import_examples.heavy_file_to_load import hello
 
print(f'{datetime.now()}: after import')
from import_examples.light_file_to_load import bye
 
 
def main():
    args: List[str] = sys.argv
    function_to_call = args[1]
    username = args[2]
    print(f'function_to_call={function_to_call}')
    if function_to_call == 'hello':
        hello(username)
    elif function_to_call == 'bye':
        bye(username)
 
 
if __name__ == '__main__':
    main()

heavy_file_to_load.py:

from time import sleep
 
print(f'Simulation of a long processing to import the module')
sleep(5)
 
 
def hello(name: str):
    print(f'hello {name}!')

light_file_to_load.py: 

def bye(name: str):
    print(f'bye {name}!')

Output with the hello process: 

C:\Users\david\AppData\Local\Programs\Python\Python310\python.exe C:/programmation/workspace-python/python_blog_examples/import_examples/lazy_import_example.py hello david
2023-04-20 17:30:44.991472: before import
Simulation of a long processing to import the module
2023-04-20 17:30:50.005932: after import
function_to_call=hello
hello david!

Output with the bye process: 

C:\Users\david\AppData\Local\Programs\Python\Python310\python.exe C:/programmation/workspace-python/python_blog_examples/import_examples/lazy_import_example.py bye david
2023-04-20 17:31:17.777202: before import
Simulation of a long processing to import the module
2023-04-20 17:31:22.789893: after import
function_to_call=bye
bye david!

Here is the main python file modified to use lazy import for the hello process: 

from datetime import datetime
 
print(f'{datetime.now()}:start')
import sys
from typing import List
 
from import_examples.light_file_to_load import bye
 
 
def main():
    args: List[str] = sys.argv
    function_to_call = args[1]
    username = args[2]
    print(f'function_to_call={function_to_call}')
    if function_to_call == 'hello':
        from import_examples.heavy_file_to_load import hello
        hello(username)
    elif function_to_call == 'bye':
        bye(username)
 
 
if __name__ == '__main__':
    main()
    print(f'{datetime.now()}:end')

Output with the hello process: 

C:\Users\david\AppData\Local\Programs\Python\Python310\python.exe C:/programmation/workspace-python/python_blog_examples/import_examples/lazy_import_example.py hello david
2023-04-20 18:00:43.925480:start
function_to_call=hello
Simulation of a long processing to import the module
hello david!
2023-04-20 18:00:48.970915:end

Output with the bye process: 

C:\Users\david\AppData\Local\Programs\Python\Python310\python.exe C:/programmation/workspace-python/python_blog_examples/import_examples/lazy_import_example.py bye david
2023-04-20 18:01:20.413444:start
function_to_call=bye
bye david!
2023-04-20 18:01:20.444443:end

Profiling the imports duration

-X importtime: show how long each import takes.

It shows module name, cumulative time (including nested imports) and self time (excluding nested imports).
Note: its output may be broken in multi-threaded application.

To apply on a specific import:
 python3 -X importtime -c 'import asyncio'

To apply on an application, here transcribe_app.py that accepts multiple parameters :
python -X importtime transcribe_app.py --default-microphone rode --language en --model whisper-1 > importtime.log 2>&1

Profile the duration to load modules and filter them to show only the longest durations:
grep -P 'import time:' importtime.log | sort -k2 -r | head -n 30
Output: 

import time: self [us] | cumulative | imported package
import time:    510468 |    2831352 | torch
import time:    185673 |     238264 |       pkg_resources
import time:    141610 |    1385819 |   torch._meta_registrations
import time:    103909 |     103909 |                                         sympy.tensor.array.array_comprehension
import time:    101093 |     128209 |       torch._refs
import time:     97598 |     127565 |           scipy.stats._continuous_distns
import time:     97153 |      97153 |       torchaudio.pipelines._source_separation_pipeline
import time:     76517 |    1073768 |         torch._prims
import time:     67080 |      98119 |               torchaudio.models.wav2vec2.components
import time:     65444 |     385013 |   librosa.core.notation
import time:     -6428 |       4606 |         yaml.parser
import time:     59968 |      59968 |                 torch.distributed.algorithms.join
import time:     35243 |      38074 |                           sympy.core.assumptions
import time:     31039 |      31039 |                 torchaudio.models.wav2vec2.wavlm_attention
import time:     28848 |    1109671 |       torch._decomp.decompositions
import time:     21317 |     119435 |             torchaudio.models.wav2vec2.model
import time:     20163 |      20163 |   torch._C
import time:     19245 |      31023 |     torchaudio._extension
import time:     -1668 |       1961 |       scipy.linalg.decomp_cholesky

common errors

problem:
TypeError: 'module' object is not callable
cause:
generally it means that the import is incorrect. we import the name of the module and not the name of the object that we would use : the function, the variable or a class.
how to fix:
suppose we have a file called Animal.py that contains the declaration of the class animal(while it may contain any other things).
here animal is a module that contains a class : the Animal class.
So to import the animal class this declaration is incorrect:
from foo import Animal
but that declaration is correct:
from foo.Animal import Animal

Ce contenu a été publié dans Non classé. Vous pouvez le mettre en favoris avec ce permalien.

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *