給大家介紹一款強大的抓包代理工具--mitmproxy
2023-11-24 06:02:30
最近工作中涉及到和app相關的測試工作，需要用到mock，特意網上查了些資料，發現有很多工具可以實現app的mock，但是經過我反覆試用後，發現mitmproxy這個工具非常的強大
我認為mitmproxy的最大優勢有2個
a、使用簡單，上手成本極低
b、可以完美的python指令碼整合，可以對http請求/websock請求做高度客製化化且自動化的處理，更重要的是，他可以熱載入
另外如果大家的工作涉及到app的測試，我強烈建議大家要使用代理抓包分析介面請求，有以下幾個優勢
a、可以比較不同平臺app的介面處理邏輯，請求頻率、觸發條件 是否一樣
b、可以觀察app的請求頻率是否合理，不合理的請求頻率會浪費客戶的流量
下面我們進入正題
一、前期準備
1、官網地址（有非常詳細的介紹，建議去看下）
https://mitmproxy.org/
2、安裝（直接使用pip安裝即可）
pip install mitmproxy
安裝後大家可以使用搜尋工具搜尋如下檔案：mitmproxy.exe\mitmweb.exe\mitmweb.exe，然後執行下面的命令，有輸出即可證明安裝成功
>mitmdump.exe --version Mitmproxy: 8.0.0 Python: 3.8.10 OpenSSL: OpenSSL 1.1.1n 15 Mar 2022 Platform: Windows-10-10.0.19045-SP0 >mitmproxy.exe --version Mitmproxy: 8.0.0 Python: 3.8.10 OpenSSL: OpenSSL 1.1.1n 15 Mar 2022 Platform: Windows-10-10.0.19045-SP0 >mitmweb.exe --version Mitmproxy: 8.0.0 Python: 3.8.10 OpenSSL: OpenSSL 1.1.1n 15 Mar 2022 Platform: Windows-10-10.0.19045-SP0
3、啟動
其實3個命令的啟動是一樣的，這裡我認為mitmweb.exe是最適合的最強大，既有web介面，也可以整合python指令碼，所以今天以介紹mitmweb.exe這個命令為主
>mitmweb.exe --help
usage: mitmweb [options]
optional arguments:
-h, --help show this help message and exit
--version show version number and exit
--options Show all options and their default values
--commands Show all commands and their signatures
--set option[=value] Set an option. When the value is omitted, booleans are set to true, strings and integers are
set to None (if permitted), and sequences are emptied. Boolean values can be true, false or
toggle. Sequences are set using multiple invocations to set for the same option.
-q, --quiet Quiet.
-v, --verbose Increase log verbosity.
--mode MODE, -m MODE Mode can be "regular", "transparent", "socks5", "reverse:SPEC", or "upstream:SPEC". For
reverse and upstream proxy modes, SPEC is host specification in the form of
"http[s]://host[:port]".
--no-anticache
--anticache Strip out request headers that might cause the server to return 304-not-modified.
--no-showhost
--showhost Use the Host header to construct URLs for display.
--rfile PATH, -r PATH
Read flows from file.
--scripts SCRIPT, -s SCRIPT
Execute a script. May be passed multiple times.
--stickycookie FILTER
Set sticky cookie filter. Matched against requests.
--stickyauth FILTER Set sticky auth filter. Matched against requests.
--save-stream-file PATH, -w PATH
Stream flows to file as they arrive. Prefix path with + to append. The full path can use
python strftime() formating, missing directories are created as needed. A new file is opened
every time the formatted string changes.
--no-anticomp
--anticomp Try to convince servers to send us un-compressed data.
Mitmweb:
--no-web-open-browser
--web-open-browser Start a browser.
--web-port PORT Web UI port.
--web-host HOST Web UI host.
--web-columns WEB_COLUMNS
Columns to show in the flow list May be passed multiple times.
Proxy Options:
--listen-host HOST Address to bind proxy to.
--listen-port PORT, -p PORT
Proxy service port.
--no-server, -n
--server Start a proxy server. Enabled by default.
--ignore-hosts HOST Ignore host and forward all traffic without processing it. In transparent mode, it is
recommended to use an IP address (range), not the hostname. In regular mode, only SSL traffic
is ignored and the hostname should be used. The supplied value is interpreted as a regular
expression and matched on the ip or the hostname. May be passed multiple times.
--allow-hosts HOST Opposite of --ignore-hosts. May be passed multiple times.
--tcp-hosts HOST Generic TCP SSL proxy mode for all hosts that match the pattern. Similar to --ignore-hosts,
but SSL connections are intercepted. The communication contents are printed to the log in
verbose mode. May be passed multiple times.
--upstream-auth USER:PASS
Add HTTP Basic authentication to upstream proxy and reverse proxy requests. Format:
username:password.
--proxyauth SPEC Require proxy authentication. Format: "username:pass", "any" to accept any user/pass
combination, "@path" to use an Apache htpasswd file, or
"ldap[s]:url_server_ldap[:port]:dn_auth:password:dn_subtree" for LDAP authentication.
--no-rawtcp
--rawtcp Enable/disable raw TCP connections. TCP connections are enabled by default.
--no-http2
--http2 Enable/disable HTTP/2 support. HTTP/2 support is enabled by default.
SSL:
--certs SPEC SSL certificates of the form "[domain=]path". The domain may include a wildcard, and is equal
to "*" if not specified. The file at path is a certificate in PEM format. If a private key is
included in the PEM, it is used, else the default key in the conf dir is used. The PEM file
should contain the full certificate chain, with the leaf certificate as the first entry. May
be passed multiple times.
--cert-passphrase PASS
Passphrase for decrypting the private key provided in the --cert option. Note that passing
cert_passphrase on the command line makes your passphrase visible in your system's process
list. Specify it in config.yaml to avoid this.
--no-ssl-insecure
--ssl-insecure, -k Do not verify upstream server SSL/TLS certificates.
--key-size KEY_SIZE TLS key size for certificates and CA.
Client Replay:
--client-replay PATH, -C PATH
Replay client requests from a saved file. May be passed multiple times.
Server Replay:
--server-replay PATH, -S PATH
Replay server responses from a saved file. May be passed multiple times.
--no-server-replay-kill-extra
--server-replay-kill-extra
Kill extra requests during replay (for which no replayable response was found).
--no-server-replay-nopop
--server-replay-nopop
Don't remove flows from server replay state after use. This makes it possible to replay same
response multiple times.
--no-server-replay-refresh
--server-replay-refresh
Refresh server replay responses by adjusting date, expires and last-modified headers, as well
as adjusting cookie expiration.
Map Remote:
--map-remote PATTERN, -M PATTERN
Map remote resources to another remote URL using a pattern of the form "[/flow-filter]/url-
regex/replacement", where the separator can be any character. May be passed multiple times.
Map Local:
--map-local PATTERN Map remote resources to a local file using a pattern of the form "[/flow-filter]/url-
regex/file-or-directory-path", where the separator can be any character. May be passed
multiple times.
Modify Body:
--modify-body PATTERN, -B PATTERN
Replacement pattern of the form "[/flow-filter]/regex/[@]replacement", where the separator can
be any character. The @ allows to provide a file path that is used to read the replacement
string. May be passed multiple times.
Modify Headers:
--modify-headers PATTERN, -H PATTERN
Header modify pattern of the form "[/flow-filter]/header-name/[@]header-value", where the
separator can be any character. The @ allows to provide a file path that is used to read the
header value string. An empty header-value removes existing header-name headers. May be passed
multiple times.
Filters:
See help in mitmproxy for filter expression syntax.
--intercept FILTER Intercept filter expression.
說實話有這麼命令引數，其實我們常用的就2個
mitmweb.exe -p 8888 -s D:\code\test\mitm4.py
-p指定代理服務的啟動埠
-s指定我們執行的指令碼
4、設定代理
正常情況下，我們app和筆電連在同一個wifi就可以代理抓包，但是在我實際測試工程中，會發現有的時候會抓不到包。所以這個時候建議大家在筆電上開熱點，然後手機連這個熱點，然後在設定代理，就基本上沒問題
5、安裝證書
由於目前的請求基本上都是https，所以我們需要安裝證書,以便mitmproxy可以幫我們解密https的報文
如果mitmweb已經啟動成功，我們就可以存取下面這個地址
http://mitm.it/
如果出現這個，就說明我們的代理還沒有設定對
如果出現下面的視窗，則證明正常，我們根據當前的使用者端型別安裝證書即可
二、進入實戰（這裡我們用51cto這個app來做測試）
1、指令碼的模板（當然還有其他勾點方法，但是基本不常用，我也沒有去使用，所以這裡不講，這裡只講request和response，其他可以看官網）
import mitmproxy.http
from mitmproxy import ctx, http
class Mit():
def request(self,flow:mitmproxy.http.HTTPFlow):
#在這裡寫對request請求的流程處理
pass
def response(self,flow:mitmproxy.http.HTTPFlow):
#在這裡寫對response請求的流程處理
pass
addons = [
Mit(),
]
1、修改請求的報文（ctx.log是proxy自己寫的一個紀錄檔工具，可以根據紀錄檔的級別在控制檯列印不同的顏色，你也可以用print列印）
a、先用下面的方法獲取請求這個物件有什麼方法
class Mit():
def request(self,flow:mitmproxy.http.HTTPFlow):
#在這裡寫對request請求的流程處理
if flow.request.url == "edu.51cto.com/app.php":
ctx.log.error("匹配到請求報文")
ctx.log.error(dir(flow.request))
def response(self,flow:mitmproxy.http.HTTPFlow):
#在這裡寫對response請求的流程處理
pass
addons = [
Mit(),
]
可以看到控制檯輸出，request這個請求有如下方法，所有大家可想而知mitmproxy這個工具有多強大了，也就是說我們幾乎可以對請求字任何欄位刪除和更改，甚至可以對新增欄位
['__abstractmethods__',
'__annotations__',
'__class__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__eq__',
'__format__',
'__ge__',
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__le__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'__weakref__',
'_abc_impl',
'_get_content_type_charset',
'_get_cookies',
'_get_multipart_form',
'_get_query',
'_get_urlencoded_form',
'_guess_encoding',
'_set_cookies',
'_set_multipart_form',
'_set_query',
'_set_urlencoded_form',
'anticache',
'anticomp',
'authority',
'constrain_encoding',
'content',
'cookies',
'copy',
'data',
'decode',
'encode',
'first_line_format',
'from_state',
'get_content',
'get_state',
'get_text',
'headers',
'host',
'host_header',
'http_version',
'is_http10',
'is_http11',
'is_http2',
'json',
'make',
'method',
'multipart_form',
'path',
'path_components',
'port',
'pretty_host',
'pretty_url',
'query',
'raw_content',
'scheme',
'set_content',
'set_state',
'set_text',
'stream',
'text',
'timestamp_end',
'timestamp_start',
'trailers',
'url',
'urlencoded_form']
b、獲取引數和修改引數的例子
我們常用用法，下面的程式碼我演示了獲取引數和修改的引數的用法，大家注意看我的註釋
def request(self,flow:mitmproxy.http.HTTPFlow):
#在這裡寫對request請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
ctx.log.error("匹配到請求報文")
# ctx.log.error(dir(flow.request))
ctx.log.error(flow.request.query)
# 返回url的引數的值
ctx.log.error(flow.request.query.items())
# 返回url引數值的dict形式
ctx.log.debug(flow.request.query.keys())
# url引數的key
ctx.log.info(flow.request.query.values())
# url引數的values值
ctx.log.info(flow.request.method)
# 獲取請求的方法
ctx.log.info(flow.request.scheme)
# 獲取請求的型別，http還是https
ctx.log.info(flow.request.host)
# 獲取請求的host
ctx.log.info(flow.request.headers)
# 獲取請求的頭部
ctx.log.info(flow.request.url)
# 獲取請求的url
flow.request.url = "https://www.baidu.com"
#修改url
flow.request.query.set_all("wd",["test"])
#設定url引數
大家看到我在app上請求51cto的地址，被代理到了百度，且新增了url的引數
c、新增引數的例子
另外，我們不僅僅可以修改已有引數，還可以給請求頭新增引數
flow.request.headers["test"] = "abc"
2、修改響應
a、先用下面的方法看下響應請求有什麼屬性和方法
class Mit():
def request(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對request請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
pass
def response(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對response請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
ctx.log.error(dir(flow.response))
addons = [
Mit(),
]
可以看到控制檯輸出，response這個請求有如下方法，所有大家可想而知mitmproxy這個工具有多強大了，也就是說我們幾乎可以對請求的任何做增刪改
['__abstractmethods__',
'__annotations__',
'__class__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__eq__',
'__format__',
'__ge__',
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__le__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'__weakref__',
'_abc_impl',
'_get_content_type_charset',
'_get_cookies',
'_guess_encoding',
'_set_cookies',
'content',
'cookies',
'copy',
'data',
'decode',
'encode',
'from_state',
'get_content',
'get_state',
'get_text',
'headers',
'http_version',
'is_http10',
'is_http11',
'is_http2',
'json',
'make',
'raw_content',
'reason',
'refresh',
'set_content',
'set_state',
'set_text',
'status_code',
'stream',
'text',
'timestamp_end',
'timestamp_start',
'trailers']
b、演示一個拒絕響應的例子
class Mit():
def request(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對request請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
pass
def response(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對response請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
ctx.log.error(dir(flow.response))
ctx.log.error("測試拒絕響應")
flow.response = flow.response.make(status_code=404,
content="<h1>被代理</h1>",
headers={"content-type":"text/html"}
)
addons = [
Mit(),
]
c、演示一個獲取引數的例子
import json
class Mit():
def request(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對request請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
pass
def response(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對response請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
res_text = json.loads(flow.response.text,encoding="utf-8")
ctx.log.info(res_text)
res_get_text = flow.response.get_text("result")
ctx.log.warn(res_get_text)
res_get_header = flow.response.headers
ctx.log.error(res_get_header)
addons = [
Mit(),
]
看一下返回的結果
flow.response.text的值，也就是響應體的值
d、演示一個修改引數的例子
import json
class Mit():
def request(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對request請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
pass
def response(self, flow: mitmproxy.http.HTTPFlow):
# 在這裡寫對response請求的流程處理
if "edu.51cto.com/app.php" in flow.request.url:
res_text = json.loads(flow.response.text,encoding="utf-8")
res_text["msg"] = "test_mitmproxy"
flow.response.text = json.dumps(res_text,ensure_ascii=True)
addons = [
Mit(),
]
我們修改了msg的資訊
至此我認為主要的作用已經講完了