Improve import performance for inference

[Yc-Chen]

What does this PR do?

Related Issue(s):
#442, #503

Type of Change

• Bug fix (non-breaking change that fixes an issue)

Testing

• I have tested this change locally
• I have added/updated tests for this change

Test details:

Added test for import performance.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 47.29730% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 31%. Comparing base (82ad5d1) to head (34fa6d7).
⚠️ Report is 21 commits behind head on develop.

❌ Your patch check has failed because the patch coverage (47%) is below the target coverage (95%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (31%) is below the target coverage (95%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (82ad5d1) and HEAD (34fa6d7). Click for more details.

HEAD has 3 uploads less than BASE

Flag	BASE (82ad5d1)	HEAD (34fa6d7)
Linux	5	4
py3.12	2	1
gpu	1	0

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #689     +/-   ##
========================================
- Coverage       60%    31%    -29%     
========================================
  Files           52     55      +3     
  Lines         6734   7058    +324     
========================================
- Hits          4041   2212   -1829     
- Misses        2693   4846   +2153     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Borda]

it seems there are to many unrelated chnages, could you pls rebase to develop 🐰