readline,readlines,read占用内存分析

110 阅读 0 评论 73 点赞

我是靠谱客的博主活力心情，这篇文章主要介绍readline,readlines,read占用内存分析，现在分享给大家，希望可以做个参考。

复制代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
原始语料：546M
1.readlines()
    with codecs.open(combine_bigram_remove_freq_1_filename, encoding='utf-8') as f:
        temp_list = [item for item in f.readlines()]
CPU usage：34.6% ==> 61.7%
占用内存：2.168G
耗时：5.66499996185s
2.
    with codecs.open(combine_bigram_remove_freq_1_filename, encoding='utf-8') as f:
        temp_list = [item for item in f]
CPU usage：35.3% ==> 62.0%
占用内存：2.136G
耗时：48.2940001488s
3.read()
    with codecs.open(combine_bigram_remove_freq_1_filename, encoding='utf-8') as f:
        temp_str =f.read()
CPU usage：30.4% ==> 36.3%
占用内存：0.48G
耗时：48.2940001488s
4.readline()
f = codecs.open(filename, encoding='utf-8')
start_time = time.time()
while 1:
    line = f.readline()
    if not line:
        end_time = time.time()
        print end_time-start_time
        break
耗时：50.6809999943