Hadoop环境的基准测试----自己电脑搭虚拟机的话就别测了，我电脑的主板差点烧了。

news/2024/5/20 1:15:39 标签: hadoop, mapreduce, hdfs

写文件测试

[user@NewBieMaster sbin]$ hadoop jar /home/user/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 8MB
2021-07-03 15:28:52,823 INFO fs.TestDFSIO: TestDFSIO.1.8
2021-07-03 15:28:52,825 INFO fs.TestDFSIO: nrFiles = 10
2021-07-03 15:28:52,825 INFO fs.TestDFSIO: nrBytes (MB) = 8.0
2021-07-03 15:28:52,825 INFO fs.TestDFSIO: bufferSize = 1000000
2021-07-03 15:28:52,825 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2021-07-03 15:28:54,350 INFO fs.TestDFSIO: creating control file: 8388608 bytes, 10 files
2021-07-03 15:28:55,845 INFO fs.TestDFSIO: created control files for: 10 files
2021-07-03 15:28:56,805 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2021-07-03 15:28:57,528 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/user/.staging/job_1625297306865_0001
2021-07-03 15:28:57,815 INFO mapred.FileInputFormat: Total input files to process : 10
2021-07-03 15:28:57,930 INFO mapreduce.JobSubmitter: number of splits:10
2021-07-03 15:28:58,471 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1625297306865_0001
2021-07-03 15:28:58,472 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-07-03 15:28:58,913 INFO conf.Configuration: resource-types.xml not found
2021-07-03 15:28:58,913 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-07-03 15:28:59,416 INFO impl.YarnClientImpl: Submitted application application_1625297306865_0001
2021-07-03 15:28:59,469 INFO mapreduce.Job: The url to track the job: http://NewBieSlave1:8088/proxy/application_1625297306865_0001/
2021-07-03 15:28:59,471 INFO mapreduce.Job: Running job: job_1625297306865_0001
2021-07-03 15:29:17,520 INFO mapreduce.Job: Job job_1625297306865_0001 running in uber mode : false
2021-07-03 15:29:17,521 INFO mapreduce.Job:  map 0% reduce 0%
2021-07-03 15:29:52,490 INFO mapreduce.Job:  map 30% reduce 0%
2021-07-03 15:29:53,505 INFO mapreduce.Job:  map 40% reduce 0%
2021-07-03 15:30:24,163 INFO mapreduce.Job:  map 53% reduce 13%
2021-07-03 15:30:25,183 INFO mapreduce.Job:  map 73% reduce 13%
2021-07-03 15:30:26,197 INFO mapreduce.Job:  map 97% reduce 13%
2021-07-03 15:30:27,215 INFO mapreduce.Job:  map 100% reduce 13%
2021-07-03 15:30:29,258 INFO mapreduce.Job:  map 100% reduce 100%
2021-07-03 15:30:29,272 INFO mapreduce.Job: Job job_1625297306865_0001 completed successfully
2021-07-03 15:30:29,413 INFO mapreduce.Job: Counters: 54
	File System Counters
		FILE: Number of bytes read=827
		FILE: Number of bytes written=2638353
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=2240
		HDFS: Number of bytes written=83886154
		HDFS: Number of read operations=45
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=12
		HDFS: Number of bytes read erasure-coded=0
	Job Counters 
		Launched map tasks=10
		Launched reduce tasks=1
		Data-local map tasks=10
		Total time spent by all maps in occupied slots (ms)=521761
		Total time spent by all reduces in occupied slots (ms)=34115
		Total time spent by all map tasks (ms)=521761
		Total time spent by all reduce tasks (ms)=34115
		Total vcore-milliseconds taken by all map tasks=521761
		Total vcore-milliseconds taken by all reduce tasks=34115
		Total megabyte-milliseconds taken by all map tasks=534283264
		Total megabyte-milliseconds taken by all reduce tasks=34933760
	Map-Reduce Framework
		Map input records=10
		Map output records=50
		Map output bytes=721
		Map output materialized bytes=881
		Input split bytes=1120
		Combine input records=0
		Combine output records=0
		Reduce input groups=5
		Reduce shuffle bytes=881
		Reduce input records=50
		Reduce output records=5
		Spilled Records=100
		Shuffled Maps =10
		Failed Shuffles=0
		Merged Map outputs=10
		GC time elapsed (ms)=9914
		CPU time spent (ms)=15330
		Physical memory (bytes) snapshot=2518188032
		Virtual memory (bytes) snapshot=30102917120
		Total committed heap usage (bytes)=2432016384
		Peak Map Physical memory (bytes)=250089472
		Peak Map Virtual memory (bytes)=2739892224
		Peak Reduce Physical memory (bytes)=141185024
		Peak Reduce Virtual memory (bytes)=2739499008
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=1120
	File Output Format Counters 
		Bytes Written=74
2021-07-03 15:30:29,634 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
2021-07-03 15:30:29,634 INFO fs.TestDFSIO:             Date & time: Sat Jul 03 15:30:29 ULAT 2021
2021-07-03 15:30:29,634 INFO fs.TestDFSIO:         Number of files: 10
2021-07-03 15:30:29,634 INFO fs.TestDFSIO:  Total MBytes processed: 80
2021-07-03 15:30:29,634 INFO fs.TestDFSIO:       Throughput mb/sec: 1.98
2021-07-03 15:30:29,634 INFO fs.TestDFSIO:  Average IO rate mb/sec: 2.11
2021-07-03 15:30:29,634 INFO fs.TestDFSIO:   IO rate std deviation: 0.57
2021-07-03 15:30:29,634 INFO fs.TestDFSIO:      Test exec time sec: 93.57
2021-07-03 15:30:29,634 INFO fs.TestDFSIO: 
[user@NewBieMaster sbin]$ hadoop jar //home/user/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 18MB
2021-07-03 15:30:42,518 INFO fs.TestDFSIO: TestDFSIO.1.8
2021-07-03 15:30:42,534 INFO fs.TestDFSIO: nrFiles = 10
2021-07-03 15:30:42,534 INFO fs.TestDFSIO: nrBytes (MB) = 18.0
2021-07-03 15:30:42,534 INFO fs.TestDFSIO: bufferSize = 1000000
2021-07-03 15:30:42,535 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2021-07-03 15:30:44,235 INFO fs.TestDFSIO: creating control file: 18874368 bytes, 10 files
2021-07-03 15:30:45,845 INFO fs.TestDFSIO: created control files for: 10 files
2021-07-03 15:30:47,086 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2021-07-03 15:30:48,403 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/user/.staging/job_1625297306865_0002
2021-07-03 15:30:49,059 INFO mapred.FileInputFormat: Total input files to process : 10
2021-07-03 15:30:49,322 INFO mapreduce.JobSubmitter: number of splits:10
2021-07-03 15:30:49,777 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1625297306865_0002
2021-07-03 15:30:49,793 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-07-03 15:30:50,233 INFO conf.Configuration: resource-types.xml not found
2021-07-03 15:30:50,233 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-07-03 15:30:50,370 INFO impl.YarnClientImpl: Submitted application application_1625297306865_0002
2021-07-03 15:30:50,438 INFO mapreduce.Job: The url to track the job: http://NewBieSlave1:8088/proxy/application_1625297306865_0002/
2021-07-03 15:30:50,440 INFO mapreduce.Job: Running job: job_1625297306865_0002
2021-07-03 15:31:04,509 INFO mapreduce.Job: Job job_1625297306865_0002 running in uber mode : false
2021-07-03 15:31:04,510 INFO mapreduce.Job:  map 0% reduce 0%
2021-07-03 15:31:34,155 INFO mapreduce.Job:  map 30% reduce 0%
2021-07-03 15:31:35,168 INFO mapreduce.Job:  map 40% reduce 0%
2021-07-03 15:31:53,392 INFO mapreduce.Job:  map 50% reduce 0%
2021-07-03 15:31:56,448 INFO mapreduce.Job:  map 80% reduce 17%
2021-07-03 15:31:57,468 INFO mapreduce.Job:  map 100% reduce 17%
2021-07-03 15:31:59,559 INFO mapreduce.Job:  map 100% reduce 100%
2021-07-03 15:32:00,583 INFO mapreduce.Job: Job job_1625297306865_0002 completed successfully
2021-07-03 15:32:00,733 INFO mapreduce.Job: Counters: 54
	File System Counters
		FILE: Number of bytes read=840
		FILE: Number of bytes written=2638379
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=2240
		HDFS: Number of bytes written=188743754
		HDFS: Number of read operations=45
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=12
		HDFS: Number of bytes read erasure-coded=0
	Job Counters 
		Launched map tasks=10
		Launched reduce tasks=1
		Data-local map tasks=10
		Total time spent by all maps in occupied slots (ms)=403783
		Total time spent by all reduces in occupied slots (ms)=21658
		Total time spent by all map tasks (ms)=403783
		Total time spent by all reduce tasks (ms)=21658
		Total vcore-milliseconds taken by all map tasks=403783
		Total vcore-milliseconds taken by all reduce tasks=21658
		Total megabyte-milliseconds taken by all map tasks=413473792
		Total megabyte-milliseconds taken by all reduce tasks=22177792
	Map-Reduce Framework
		Map input records=10
		Map output records=50
		Map output bytes=734
		Map output materialized bytes=894
		Input split bytes=1120
		Combine input records=0
		Combine output records=0
		Reduce input groups=5
		Reduce shuffle bytes=894
		Reduce input records=50
		Reduce output records=5
		Spilled Records=100
		Shuffled Maps =10
		Failed Shuffles=0
		Merged Map outputs=10
		GC time elapsed (ms)=6023
		CPU time spent (ms)=11220
		Physical memory (bytes) snapshot=2538414080
		Virtual memory (bytes) snapshot=30104772608
		Total committed heap usage (bytes)=2432016384
		Peak Map Physical memory (bytes)=248365056
		Peak Map Virtual memory (bytes)=2740322304
		Peak Reduce Physical memory (bytes)=140070912
		Peak Reduce Virtual memory (bytes)=2739499008
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=1120
	File Output Format Counters 
		Bytes Written=74
2021-07-03 15:32:00,964 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
2021-07-03 15:32:00,964 INFO fs.TestDFSIO:             Date & time: Sat Jul 03 15:32:00 ULAT 2021
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:         Number of files: 10
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:  Total MBytes processed: 180
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:       Throughput mb/sec: 4.82
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:  Average IO rate mb/sec: 5.28
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:   IO rate std deviation: 2.16
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:      Test exec time sec: 74.89
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:

读文件测试

[user@NewBieMaster sbin]$ hadoop jar //home/user/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 18MB
2021-07-03 15:30:42,518 INFO fs.TestDFSIO: TestDFSIO.1.8
2021-07-03 15:30:42,534 INFO fs.TestDFSIO: nrFiles = 10
2021-07-03 15:30:42,534 INFO fs.TestDFSIO: nrBytes (MB) = 18.0
2021-07-03 15:30:42,534 INFO fs.TestDFSIO: bufferSize = 1000000
2021-07-03 15:30:42,535 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2021-07-03 15:30:44,235 INFO fs.TestDFSIO: creating control file: 18874368 bytes, 10 files
2021-07-03 15:30:45,845 INFO fs.TestDFSIO: created control files for: 10 files
2021-07-03 15:30:47,086 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2021-07-03 15:30:48,403 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/user/.staging/job_1625297306865_0002
2021-07-03 15:30:49,059 INFO mapred.FileInputFormat: Total input files to process : 10
2021-07-03 15:30:49,322 INFO mapreduce.JobSubmitter: number of splits:10
2021-07-03 15:30:49,777 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1625297306865_0002
2021-07-03 15:30:49,793 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-07-03 15:30:50,233 INFO conf.Configuration: resource-types.xml not found
2021-07-03 15:30:50,233 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-07-03 15:30:50,370 INFO impl.YarnClientImpl: Submitted application application_1625297306865_0002
2021-07-03 15:30:50,438 INFO mapreduce.Job: The url to track the job: http://NewBieSlave1:8088/proxy/application_1625297306865_0002/
2021-07-03 15:30:50,440 INFO mapreduce.Job: Running job: job_1625297306865_0002
2021-07-03 15:31:04,509 INFO mapreduce.Job: Job job_1625297306865_0002 running in uber mode : false
2021-07-03 15:31:04,510 INFO mapreduce.Job:  map 0% reduce 0%
2021-07-03 15:31:34,155 INFO mapreduce.Job:  map 30% reduce 0%
2021-07-03 15:31:35,168 INFO mapreduce.Job:  map 40% reduce 0%
2021-07-03 15:31:53,392 INFO mapreduce.Job:  map 50% reduce 0%
2021-07-03 15:31:56,448 INFO mapreduce.Job:  map 80% reduce 17%
2021-07-03 15:31:57,468 INFO mapreduce.Job:  map 100% reduce 17%
2021-07-03 15:31:59,559 INFO mapreduce.Job:  map 100% reduce 100%
2021-07-03 15:32:00,583 INFO mapreduce.Job: Job job_1625297306865_0002 completed successfully
2021-07-03 15:32:00,733 INFO mapreduce.Job: Counters: 54
	File System Counters
		FILE: Number of bytes read=840
		FILE: Number of bytes written=2638379
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=2240
		HDFS: Number of bytes written=188743754
		HDFS: Number of read operations=45
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=12
		HDFS: Number of bytes read erasure-coded=0
	Job Counters 
		Launched map tasks=10
		Launched reduce tasks=1
		Data-local map tasks=10
		Total time spent by all maps in occupied slots (ms)=403783
		Total time spent by all reduces in occupied slots (ms)=21658
		Total time spent by all map tasks (ms)=403783
		Total time spent by all reduce tasks (ms)=21658
		Total vcore-milliseconds taken by all map tasks=403783
		Total vcore-milliseconds taken by all reduce tasks=21658
		Total megabyte-milliseconds taken by all map tasks=413473792
		Total megabyte-milliseconds taken by all reduce tasks=22177792
	Map-Reduce Framework
		Map input records=10
		Map output records=50
		Map output bytes=734
		Map output materialized bytes=894
		Input split bytes=1120
		Combine input records=0
		Combine output records=0
		Reduce input groups=5
		Reduce shuffle bytes=894
		Reduce input records=50
		Reduce output records=5
		Spilled Records=100
		Shuffled Maps =10
		Failed Shuffles=0
		Merged Map outputs=10
		GC time elapsed (ms)=6023
		CPU time spent (ms)=11220
		Physical memory (bytes) snapshot=2538414080
		Virtual memory (bytes) snapshot=30104772608
		Total committed heap usage (bytes)=2432016384
		Peak Map Physical memory (bytes)=248365056
		Peak Map Virtual memory (bytes)=2740322304
		Peak Reduce Physical memory (bytes)=140070912
		Peak Reduce Virtual memory (bytes)=2739499008
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=1120
	File Output Format Counters 
		Bytes Written=74
2021-07-03 15:32:00,964 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
2021-07-03 15:32:00,964 INFO fs.TestDFSIO:             Date & time: Sat Jul 03 15:32:00 ULAT 2021
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:         Number of files: 10
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:  Total MBytes processed: 180
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:       Throughput mb/sec: 4.82
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:  Average IO rate mb/sec: 5.28
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:   IO rate std deviation: 2.16
2021-07-03 15:32:00,965 INFO fs.TestDFSIO:      Test exec time sec: 74.89
2021-07-03 15:32:00,965 INFO fs.TestDFSIO: 
[user@NewBieMaster sbin]$ hadoop jar //home/user/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 18MB^C
[user@NewBieMaster sbin]$ hadoop jar /home/user/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB
2021-07-03 15:35:30,860 INFO fs.TestDFSIO: TestDFSIO.1.8
2021-07-03 15:35:30,893 INFO fs.TestDFSIO: nrFiles = 10
2021-07-03 15:35:30,893 INFO fs.TestDFSIO: nrBytes (MB) = 128.0
2021-07-03 15:35:30,893 INFO fs.TestDFSIO: bufferSize = 1000000
2021-07-03 15:35:30,894 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2021-07-03 15:35:32,704 INFO fs.TestDFSIO: creating control file: 134217728 bytes, 10 files
2021-07-03 15:35:34,094 INFO fs.TestDFSIO: created control files for: 10 files
2021-07-03 15:35:35,001 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2021-07-03 15:35:35,801 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/user/.staging/job_1625297306865_0003
2021-07-03 15:35:36,122 INFO mapred.FileInputFormat: Total input files to process : 10
2021-07-03 15:35:36,282 INFO mapreduce.JobSubmitter: number of splits:10
2021-07-03 15:35:37,023 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1625297306865_0003
2021-07-03 15:35:37,024 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-07-03 15:35:37,397 INFO conf.Configuration: resource-types.xml not found
2021-07-03 15:35:37,398 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-07-03 15:35:37,551 INFO impl.YarnClientImpl: Submitted application application_1625297306865_0003
2021-07-03 15:35:37,619 INFO mapreduce.Job: The url to track the job: http://NewBieSlave1:8088/proxy/application_1625297306865_0003/
2021-07-03 15:35:37,632 INFO mapreduce.Job: Running job: job_1625297306865_0003
2021-07-03 15:35:51,287 INFO mapreduce.Job: Job job_1625297306865_0003 running in uber mode : false
2021-07-03 15:35:51,289 INFO mapreduce.Job:  map 0% reduce 0%
2021-07-03 15:36:07,688 INFO mapreduce.Job:  map 20% reduce 0%
2021-07-03 15:36:26,932 INFO mapreduce.Job:  map 20% reduce 7%
2021-07-03 15:36:40,123 INFO mapreduce.Job:  map 40% reduce 7%
2021-07-03 15:36:41,133 INFO mapreduce.Job:  map 90% reduce 7%
2021-07-03 15:36:42,152 INFO mapreduce.Job:  map 100% reduce 7%
2021-07-03 15:36:43,161 INFO mapreduce.Job:  map 100% reduce 100%
2021-07-03 15:36:43,188 INFO mapreduce.Job: Job job_1625297306865_0003 completed successfully
2021-07-03 15:36:43,310 INFO mapreduce.Job: Counters: 54
	File System Counters
		FILE: Number of bytes read=830
		FILE: Number of bytes written=2638337
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=188745920
		HDFS: Number of bytes written=78
		HDFS: Number of read operations=55
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
		HDFS: Number of bytes read erasure-coded=0
	Job Counters 
		Launched map tasks=10
		Launched reduce tasks=1
		Data-local map tasks=10
		Total time spent by all maps in occupied slots (ms)=395532
		Total time spent by all reduces in occupied slots (ms)=32686
		Total time spent by all map tasks (ms)=395532
		Total time spent by all reduce tasks (ms)=32686
		Total vcore-milliseconds taken by all map tasks=395532
		Total vcore-milliseconds taken by all reduce tasks=32686
		Total megabyte-milliseconds taken by all map tasks=405024768
		Total megabyte-milliseconds taken by all reduce tasks=33470464
	Map-Reduce Framework
		Map input records=10
		Map output records=50
		Map output bytes=724
		Map output materialized bytes=884
		Input split bytes=1120
		Combine input records=0
		Combine output records=0
		Reduce input groups=5
		Reduce shuffle bytes=884
		Reduce input records=50
		Reduce output records=5
		Spilled Records=100
		Shuffled Maps =10
		Failed Shuffles=0
		Merged Map outputs=10
		GC time elapsed (ms)=7255
		CPU time spent (ms)=9130
		Physical memory (bytes) snapshot=2612715520
		Virtual memory (bytes) snapshot=30078283776
		Total committed heap usage (bytes)=2432016384
		Peak Map Physical memory (bytes)=249483264
		Peak Map Virtual memory (bytes)=2740408320
		Peak Reduce Physical memory (bytes)=142974976
		Peak Reduce Virtual memory (bytes)=2739482624
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=1120
	File Output Format Counters 
		Bytes Written=78
2021-07-03 15:36:43,409 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
2021-07-03 15:36:43,409 INFO fs.TestDFSIO:             Date & time: Sat Jul 03 15:36:43 ULAT 2021
2021-07-03 15:36:43,409 INFO fs.TestDFSIO:         Number of files: 10
2021-07-03 15:36:43,409 INFO fs.TestDFSIO:  Total MBytes processed: 180
2021-07-03 15:36:43,409 INFO fs.TestDFSIO:       Throughput mb/sec: 43.09
2021-07-03 15:36:43,409 INFO fs.TestDFSIO:  Average IO rate mb/sec: 49.26
2021-07-03 15:36:43,409 INFO fs.TestDFSIO:   IO rate std deviation: 19.99
2021-07-03 15:36:43,409 INFO fs.TestDFSIO:      Test exec time sec: 69.21
2021-07-03 15:36:43,409 INFO fs.TestDFSIO: 
[user@NewBieMaster sbin]$

用sort程序测试MR的运算速度

hadoop jar /home/user/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar randomwriter random-data
hadoop jar /home/user/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar sort random-data sorted-data


http://www.niftyadmin.cn/n/876901.html

相关文章

sql附加服务器数据库文件,批量附加sql数据库

sql附加服务器数据库文件,批量附加sql数据库

批量附加sql数据库是为数据量管理人员推出的一款sql附加数据工具，主要用于将主程序库重新安装到新的服务器上面，支持批量附加，大大提高SQL数据库管理效率，需要的朋友可以来本站下载！附加数据库提示823sql 2000 数据库附…
阅读更多...
ftp服务器目录创建文件夹权限设置,ftp服务器 创建文件夹权限设置

ftp服务器目录创建文件夹权限设置,ftp服务器 创建文件夹权限设置

ftp服务器 创建文件夹权限设置 内容精选换一换文件系统创建完成后，用户需在客户端挂载文件共享，在本地共享目录和CSG的文件共享建立映射，通过操作本地的目录实现对CSG共享目录的操作，实现数据实时上云管理。本章节主要介绍在Windo…
阅读更多...
服务器系统要设置多少位密码,服务器系统设置新密码是多少

服务器系统要设置多少位密码,服务器系统设置新密码是多少

服务器系统设置新密码是多少 内容精选换一换为保证安全，建议您定期(例如，90天)重置连接redis服务的密码，连接redis服务的默认密码是Huawei12#$。其中，dir表示Profiling的安装路径，~需替换为Mind Studio安装用户的家目录…
阅读更多...
光猫 虚拟服务器设置,光猫虚拟服务器设置

光猫 虚拟服务器设置,光猫虚拟服务器设置

光猫虚拟服务器设置 内容精选换一换对于不同的硬件设备，通过在BIOS中设置一些高级选项，可以有效提升服务器性能。此优化项只在非虚拟化场景使用，在虚拟化场景，则开启SMMU。重启服务器过程中，单击Delete键进入BIOS&…
阅读更多...
orecol数据库查询服务器信息,orecol数据库和mysql

orecol数据库查询服务器信息,orecol数据库和mysql

orecol数据库和mysql 内容精选换一换本文档以测试MySQL数据库为例,Mariadb、Percona测试步骤与MySQL相同。登录MySQL数据库。/usr/local/mysql/bin/mysql -uroot -p123456 -S /data/mysql/run/mysql.sock创建数据库sysbench。create database sysbench;sh华为云帮助中心&#x…
阅读更多...
Flume的简介以及安装配置

Flume的简介以及安装配置

简介 Flume最初是Cloudera开发的，基于流式架构，灵活简单的，高可用高可靠的 实时日志采集，聚合和传输的系统，2011年10月22日Cloudera对Flume进行了里程碑式的改动，重构核心组件，核心配置以及代码…
阅读更多...
分布式系统监控软件Ganglia的安装和配置

分布式系统监控软件Ganglia的安装和配置

1）在66服务器（主服务器）上下载安装监控软件Ganglia。 yum install -y epel-release ganglia-gmetad ganglia-devel ganglia-gmond rrdtool httpd ganglia-web php 4）在88和99服务器（非主服务器）上下载安装…
阅读更多...
Kafka Manager的简介以及安装使用

Kafka Manager的简介以及安装使用

简介 Kafka Manager 是 Yahoo 推出的 Kafka 管理工具，支持： 管理多个集群 轻松检查集群状态 (topics, brokers, replica distribution, partition distribution) 执行复制选举 生成分区指派，基于集群的状态 分区的重新指…
阅读更多...
最新文章

Copyright @ 2022~2023