Celery中使用importlib按需导包

最近在写扫描器的时候发现如果写死在run函数里边是很不明智的
这样的话每次都要将celery重启才可以生效,就想到了说可以不可以使用importlib来动态的导入
如此一来后期的插件添加时就不再需要重启celery了.

但是遇到了坑, 很容易就出现ImportError: No module named folder.script1或者TypeError: relative imports require the 'package' argument

我愁呀, 我一缕一缕的揪头发呀。

后来在celery issue上看到了相同的问题https://github.com/celery/celery/issues/3441

具体来说就是在tasks.py的路径加入了sys.path, 这样在找module的时候可以找的到该路径.
成功的代码如下

首先看tree

1
2
3
4
5
6
7
8
9
10
11
12
☁ tmp [master] ⚡ tree
.
├── __init__.py
├── folder
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── script1.py
│   ├── script1.pyc
│   ├── script2.py
│   └── script2.pyc
├── tasks.py
└── tasks.pyc

tasks.py内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#!/usr/bin/env python
# coding=utf-8
from os import path
from os import walk
from celery import Celery
import importlib
import time
import sys
import os
sys.path.append(os.getcwd()) # 这一行来将文件所在目录加入到sys.path中去
here = path.split(path.abspath(__file__))[0]
script_folder = "{0}/folder".format(here)
app = Celery("tasks", broker="redis://localhost:6379/0")
@app.task
def doWork():
print script_folder
_, _, files = walk(script_folder, True).next()
#print files
script_list = []
for f in files:
f = str(f)
if f.endswith(".py") and not f.startswith("_"):
f = f[:-3]
script_list.append(path.join(script_folder, f))
#print script_list
for script in script_list:
#print "script:\t" + script
script = ".".join(script.split("/"))[script.index("folder"):]
module = importlib.import_module(script, __name__)
t = getattr(module, "test")
tt = t("http://iqiyi.com")
result = tt.runFuzz()
print script + "[RESULT]" + str(result)
time.sleep(5)
if __name__ == '__main__':
doWork()

测试结果:
ipython处:

1
2
3
4
In [1]: from tasks import doWork
In [2]: doWork.delay()
Out[2]: <AsyncResult: 023253fc-722b-4433-9d8c-f9e6a76f988b>

celery启动处:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
-------------- celery@xxxxxxxxxxMacBook-Pro-7.local v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Darwin-15.4.0-x86_64-i386-64bit 2017-08-27 21:31:56
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: tasks:0x104f225d0
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. tasks.doWork
[2017-08-27 21:31:57,109: INFO/MainProcess] Connected to redis://localhost:6379/0
[2017-08-27 21:31:57,119: INFO/MainProcess] mingle: searching for neighbors
[2017-08-27 21:31:58,149: INFO/MainProcess] mingle: all alone
[2017-08-27 21:31:58,162: INFO/MainProcess] celery@xxxxxxxxxxxMacBook-Pro-7.local ready.
[2017-08-27 21:32:11,311: INFO/MainProcess] Received task: tasks.doWork[023253fc-722b-4433-9d8c-f9e6a76f988b]
[2017-08-27 21:32:11,315: WARNING/ForkPoolWorker-3] /Users/xxxxxxxxx/Desktop/tools/tmp/folder
[2017-08-27 21:32:11,920: WARNING/ForkPoolWorker-3] folder.script1[RESULT]200
[2017-08-27 21:32:16,922: WARNING/ForkPoolWorker-3] folder.script2[RESULT]in script2.py, scan url:http://iqiyi.com
[2017-08-27 21:32:21,924: INFO/ForkPoolWorker-3] Task tasks.doWork[023253fc-722b-4433-9d8c-f9e6a76f988b] succeeded in 10.60855711s: None
^C
worker: Hitting Ctrl+C again will terminate all running tasks!

文章目录
|