forked from apache/tez
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGES.txt
1900 lines (1838 loc) · 128 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Apache Tez Change Log
=====================
Release 0.8.3: Unreleased
INCOMPATIBLE CHANGES
TEZ-3029. Add an onError method to service plugin contexts.
ALL CHANGES:
TEZ-3189. Pre-warm dags should not be counted in submitted dags count by DAGAppMaster.
TEZ-2967. Vertex start time should be that of first task start time in UI
TEZ-3175. Add tez client submit host
TEZ-3166. Fix a few cases where counters aren't fully updated and sent for failed tasks.
TEZ-2958. Recovered TA, whose commit cannot be recovered, should move to killed state
TEZ-2936. Create ATS implementation that enables support for YARN-4265 (ATS v1.5)
TEZ-3148. Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt.
TEZ-3105. TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures.
TEZ-3155. Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits.
TEZ-2863. Container, node, and logs not available in UI for tasks that fail to launch
TEZ-3140. Reduce AM memory usage during serialization
TEZ-2756. MergeManager close should not try merging files on close if invoked after a shuffle exception.
TEZ-3156. Tez client keeps trying to talk to RM even if RM does not know about the application.
TEZ-3115. Shuffle string handling adds significant memory overhead
TEZ-3151. Expose DAG credentials to plugins.
TEZ-3149. Tez-tools: Add username in DagInfo.
TEZ-2988. DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier.
TEZ-3147. Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory
TEZ-3141. mapreduce.task.timeout is not translated to container heartbeat timeout
TEZ-3128. Avoid stopping containers on the AM shutdown thread.
TEZ-3129. Tez task and task attempt UI needs application fails with NotFoundException
TEZ-3114. Shuffle OOM due to EventMetaData flood
TEZ-1911. MergeManager's unconditionalReserve() should check for memory limits before allocating.
TEZ-3102. Fetch failure of a speculated task causes job hang
TEZ-3124. Running task hangs due to missing event to initialize input in recovery.
TEZ-3135. tez-ext-service-tests, tez-plugins/tez-yarn-timeline-history and tez-tools/tez-javadoc-tools missing dependencies.
TEZ-3134. tez-dag should depend on commons-collections4.
TEZ-3126. Log reason for not reducing parallelism
TEZ-3131. Support a way to override test_root_dir for FaultToleranceTestRunner.
TEZ-3067. Links to tez configs documentation should be bubbled up to top-level release page.
TEZ-3123. Containers can get re-used even with conflicting local resources.
TEZ-3117. Deadlock in Edge and Vertex code
TEZ-3103. Shuffle can hang when memory to memory merging enabled
TEZ-3107. tez-tools: Log warn msgs in case ATS has wrong values (e.g startTime > finishTime).
TEZ-3104. Tez fails on Bzip2 intermediate output format on hadoop 2.7.1 and earlier
TEZ-3090. MRInput should make dagIdentifier, vertexIdentifier, etc available to the InputFormat jobConf.
TEZ-3093. CriticalPathAnalyzer should be accessible via zeppelin.
TEZ-3089. TaskConcurrencyAnalyzer can return negative task count with very large jobs.
TEZ-2307. Possible wrong error message when submitting new dag
TEZ-2974. Tez tools: TFileRecordReader in tez-tools should support reading >2 GB tfiles.
TEZ-3081. Update tez website for trademarks feedback.
TEZ-3076. Reduce merge memory overhead to support large number of in-memory mapoutputs
TEZ-3079. Fix tez-tfile parser documentation.
TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services.
TEZ-3036. Tez AM can hang on startup with no indication of error
TEZ-3052. Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED
TEZ-2594. Fix LICENSE for missing entries for full and minimal tarballs.
TEZ-3053. Containers timeout if they do not receive a task within the container timeout interval.
TEZ-2898. tez tools : swimlanes.py is broken.
TEZ-2937. Can Processor.close() be called after closing inputs and outputs?
TEZ-3037. History URL should be set regardless of which history logging service is enabled.
TEZ-3032. DAG start time getting logged using system time instead of recorded time in startTime field.
TEZ-3101. Tez UI: Task attempt log link doesn't have the correct protocol.
TEZ-3143. Tez UI 2: Make the build faster
TEZ-2980: Tez UI 2 - Umbrella:
TEZ-2982. Tez UI: Create tez-ui2 directory and get a basic dummy page working in ember 2.2
TEZ-3016. Tez UI 2: Make bower dependency silent
TEZ-2983. Tez UI 2: Get ember initializers functional
TEZ-3018. Tez UI 2: Add config.env
TEZ-3019. Tez UI 2: Replace BaseURL with Host
TEZ-2984. Tez UI 2: Create abstract classes
TEZ-3020. Tez UI 2: Add entity blueprint
TEZ-2985. Tez UI 2: Create loader and entity classes
TEZ-3021. Tez UI 2: Add env service & initializer
TEZ-3023. Tez UI 2: Abstract adapter and route
TEZ-3022. Tez UI 2: Add serializer & adapter for timeline server
TEZ-3026. Tez UI 2: Add adapters for RM & AM
TEZ-3027. Tez UI 2: Add header and footer elements
TEZ-2986. Tez UI 2: Implement All DAGs page
TEZ-3038. Tez UI 2: Create DAG details page
TEZ-3039. Tez UI 2: Create all sub-pages for DAG
TEZ-3040. Tez UI 2: Create Vertex details page & sub tables
TEZ-3041. Tez UI 2: Create Task & Attempt details page with sub tables
TEZ-3045. Tez UI 2: Create application details page with DAGs tab
TEZ-3048. Tez UI 2: Make PhantomJS a local dependency for build tests
TEZ-3042. Tez UI 2: Create Counters pages
TEZ-3043. Tez UI 2: Create configurations page
TEZ-3049. Tez UI 2: Add column selector
TEZ-3050. Tez UI 2: Add counter columns
TEZ-3064. Tez UI 2: Add All DAGs filters
TEZ-3059. Tez UI 2: Make refresh functional
TEZ-3070. Tez UI 2: Jenkins build is failing
TEZ-3060. Tez UI 2: Activate auto-refresh
TEZ-3061. Tez UI 2: Display in-progress vertex table in DAG details
TEZ-3069. Tez UI 2: Make error bar fully functional
TEZ-3062. Tez UI 2: Integrate graphical view
TEZ-3058. Tez UI 2: Add download data functionality
TEZ-3084. Tez UI 2: Display caller type and info
TEZ-3080. Tez UI 2: Ensure UI 2 is in-line with UI 1
TEZ-3092. Tez UI 2: Tuneups & Improvements
TEZ-3095. Tez UI 2: Tuneups & Improvements
TEZ-3088. Tez UI 2: Licenses of all the packages used by Tez Ui must be documented
TEZ-2916. Tez UI 2: Show counts of running tasks on the DAG visualization page
TEZ-3125. Tez UI 2: All auto-refresh pages refresh multiple times shortly after application complete
TEZ-3127. Tez UI 2: Release audit is failing
Release 0.8.2: 2016-01-19
INCOMPATIBLE CHANGES
TEZ-3024. Move TaskCommunicator to correct package.
TEZ-2679. Admin forms of launch env settings
TEZ-2948. Stop using dagName in the dagComplete notification to TaskCommunicators.
TEZ-2949. Allow duplicate dag names within session for Tez.
TEZ-604. Revert temporary changes made in TEZ-603 to kill the provided tez session, if running a MapReduce job.
TEZ-2972. Avoid task rescheduling when a node turns unhealthy
ALL CHANGES:
TEZ-2669. Propagation of errors from plugins to the AM for error reporting.
TEZ-2978. Add an option to allow the SplitGrouper to generate node local only groups.
TEZ-2129. Task and Attempt views should contain links to the logs
TEZ-3025. InputInitializer creation should use the dag ugi.
TEZ-3017. HistoryACLManager does not have a close method for cleanup
TEZ-2914. Ability to limit vertex concurrency
TEZ-3011. Link Vertex Name in Dag Tasks/Task Attempts to Vertex
TEZ-3006. Remove unused import in TestHistoryParser.
TEZ-2910. Set caller context for tracing ( integrate with HDFS-9184 ).
TEZ-2976. Recovery fails when InputDescriptor is changed during input initialization.
TEZ-2997. Tez UI: Support searches by CallerContext ID for DAGs
TEZ-2996. TestAnalyzer fails in trunk after recovery redesign
TEZ-2987. TestVertexImpl.testTez2684 fails
TEZ-2995. Timeline primary filter should only be on callerId and not type.
TEZ-2994. LocalProgress in tez-runtime-library missing Apache header, rat check warnings from the new licenses after TEZ-2592 merge.
TEZ-2977. Make HadoopShim selection be overridable for distro-specific implementations.
TEZ-2472. Change slf4j version to 1.7.10.
TEZ-2920. org.apache.tez.client.TestTezClient.testStopRetriesUntilTimeout is flaky.
TEZ-2824. Add javadocs for Vertex.setConf and DAG.setConf.
TEZ-2911. Null location Strings can cause problems with GroupedSplit serialization.
TEZ-2990. Change test-patch.sh to run through all tests, despite failures in upstream modules
TEZ-2798. NPE when executing TestMemoryWithEvents::testMemoryScatterGather.
TEZ-2963. RecoveryService#handleSummaryEvent exception with HDFS transparent encryption + kerberos authentication.
TEZ-2966. Tez does not honor mapreduce.task.timeout
TEZ-2979. FlakyTest: org.apache.tez.history.TestHistoryParser.
TEZ-1491. Tez reducer-side merge's counter update is slow.
TEZ-2943. Change shuffle vertex manager to use per vertex data for auto
reduce and slow start
TEZ-2346. TEZ-UI: Lazy load other info / counter data
TEZ-2975. Bump up apache commons dependency.
TEZ-2970. Re-localization in TezChild does not use correct UGI.
TEZ-2968. Counter limits exception causes AM to crash.
TEZ-2960. Tez UI: Move hardcoded url namespace to the configuration file
TEZ-2581. Umbrella for Tez Recovery Redesign
TEZ-2956. Handle auto-reduce parallelism when the
totalNumBipartiteSourceTasks is 0
TEZ-2947. Tez UI: Timeline, RM & AM requests gets into a consecutive loop in counters page without any delay
TEZ-2946. Tez UI: At times RM return a huge error message making the yellow error bar to fill the whole screen
TEZ-2949. Allow duplicate dag names within session for Tez.
TEZ-2952. NPE in TestOnFileUnorderedKVOutput
TEZ-2480. Exception when closing output is ignored.
TEZ-2944. NPE in TestProcessorContext.
TEZ-2948. Stop using dagName in the dagComplete notification to TaskCommunicators.
TEZ-2945. TEZ-2740 addendum to update API with currently supported parameters
TEZ-2933. Tez UI: Load application details from RM when available
TEZ-2908. Tez UI: Errors are logged, but not displayed in the UI when AM fetch fails
TEZ-2923. Tez Live UI counters view empty for vertices, tasks, attempts
TEZ-2924. Framework for Hadoop shims.
TEZ-2935. Add MR slow start translation for ShuffleVertexManager
TEZ-2918. Make progress notifications in IOs
TEZ-2940. Invalid shuffle max slow start setting causes vertex to hang indefinitely
TEZ-2930. Tez UI: Parent controller is not polling at times
TEZ-1670. Add tests for all converter functions in HistoryEventTimelineConversion.
TEZ-2879. While grouping splits, allow an alternate list of preferred locations to be provided per split.
TEZ-2929. Tez UI: Dag details page displays vertices to be running even when dag have completed
TEZ-1976. Findbug warning: Unread field:
org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat$SplitHolder.split
TEZ-2927. Tez UI: Graciously fail when system-metrics-publisher is disabled
TEZ-2915. Tez UI: Getting back to the DAG details page is difficult
TEZ-2895. Tez UI: Add option to enable and disable in-progress
TEZ-2894. Tez UI: Disable sorting for few columns while in progress. Display an alert on trying to sort them
TEZ-2893. Tez UI: Retain vertex info displayed in DAG details page even after completion
TEZ-2878. Tez UI: AM error handling - Make the UI handle cases in which AM returns unexpected/no data
TEZ-2922. Tez Live UI gives access denied for admins
TEZ-2849. Implement Specific Workaround for JDK-8026049 & JDK-8073093.
TEZ-2828. Fix typo in "Shuffle assigned " log statement in shuffle.orderedgrouped.Shuffle.
TEZ-2909. Tez UI: Application link in All DAGs table is disable when applicationhistory is unavailable
TEZ-808. Handle task attempts that are not making progress
TEZ-2553. Tez UI: Tez UI Nits
TEZ-2814. ATSImportTool has a return statement in a finally block
TEZ-2906. Compilation fails with hadoop 2.2.0
TEZ-2900. Ignore V_INPUT_DATA_INFORMATION when vertex is in Failed/Killed/Error
TEZ-2244. PipelinedSorter: Progressive allocation for sort-buffers
TEZ-2904. Pig can't specify task specific command opts
TEZ-2888. Make critical path calculation resilient to AM crash
TEZ-2899. Tez UI: DAG getting created with huge horizontal gap in between vertices
TEZ-2907. NPE in IFile.Reader.getLength during final merge operation
TEZ-2903. Stop using proprietary APIs in RPCLoadGen.
TEZ-2882. Consider improving fetch failure handling
TEZ-2850. Tez MergeManager OOM for small Map Outputs
TEZ-1888. Fix javac warnings all over codebase.
TEZ-2886. Ability to merge AM credentials with DAG credentials.
TEZ-2896. Fix thread names used during Input/Output initialization.
TEZ-2866. Tez UI: Newly added columns wont be displayed by default in tables
TEZ-2887. Tez build failure due to missing dependency in pom files.
TEZ-1692. Reduce code duplication between TezMapredSplitsGrouper and TezMapreduceSplitsGrouper.
TEZ-2972. Avoid task rescheduling when a node turns unhealthy
Release 0.8.1-alpha: 2015-10-12
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-2885. Remove counter logs from AMWebController
TEZ-2096. TEZ-UI : Add link to view AM log of finished & running apps
TEZ-2874. Improved logging for caller context.
TEZ-2875. Enable missing tests in TestAnalyzer
TEZ-2781. Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails
TEZ-1788. Allow vertex level disabling of speculation
TEZ-2868. Fix setting Caller Context in Tez Examples.
TEZ-2860. NPE in DAGClientImpl.
TEZ-2855. Fix a potential NPE while routing VertexManager events.
TEZ-2758. Remove append API in RecoveryService after TEZ-1909.
TEZ-2851. Support a way for upstream applications to pass in a caller context to Tez.
TEZ-2859. TestMergeManager.testLocalDiskMergeMultipleTasks failing
TEZ-2858. Stop using System.currentTimeMillis in TestInputReadyTracker.
TEZ-2857. Fix flakey tests in TestDAGImpl.
TEZ-2836. Avoid setting framework/system counters for tasks running in threads.
TEZ-2398. Flaky test: TestFaultTolerance
TEZ-2833. Dont create extra directory during ATS file download
TEZ-2834. Make Tez preemption resilient to incorrect free resource reported
by YARN
TEZ-2775. Improve and consolidate logging in Runtime components.
TEZ-2097. TEZ-UI Add dag logs backend support
TEZ-2812. Preemption sometimes does not respect heartbeats between preemptions
TEZ-814. Improve heuristic for determining a task has failed outputs
TEZ-2832. Support tests for both SimpleHistory logging and ATS logging
TEZ-2827. Increase timeout for TestFetcher testInputAttemptIdentifierMap
TEZ-2774. Improvements and cleanup of logging for the AM and parts of the runtme.
TEZ-2825. Report progress in terms of completed tasks to reduce load on AM for Tez UI
TEZ-2812. Tez UI: Update task & attempt tables while in progress.
TEZ-2786. Tez UI: Update vertex, task & attempt details page while in progress.
TEZ-2612. Support for showing allocation delays due to internal preemption
TEZ-2808. Race condition between preemption and container assignment
TEZ-2807. Log data in the finish event instead of the start event
TEZ-2799. SimpleHistoryParser NPE
TEZ-2643. Minimize number of empty spills in Pipelined Sorter
TEZ-2783. Refactor analyzers to extend TezAnalyzerBase
TEZ-2784. optimize TaskImpl.isFinished()
TEZ-2788. Allow TezAnalyzerBase to parse SimpleHistory logs
TEZ-2782. VertexInfo.getAvgExecutionTimeInterval throws NPE when task does not have any valid attempts info
TEZ-2778. Improvements to handle multiple read errors with complex DAGs
TEZ-2768. Log a useful error message when the summary stream cannot be closed when shutting
down an AM.
TEZ-2745. ClassNotFoundException of user code should fail dag
TEZ-2754. Tez UI: StartTime & EndTime is not displayed with right format in Graphical View
TEZ-2752. logUnsuccessful completion in Attempt should write original finish
time to ATS
TEZ-2755. Fix findbugs warning in TezClient
TEZ-2767. Make TezMxBeanResourceCalculator the default resource calculator.
TEZ-2765. Change Xmlwriter to use defaultValue instead of value tag.
TEZ-2750. Shuffle may not shutdown in case of a fetch failure, causing it to hang.
TEZ-2294. Add tez-site-template.xml with description of config properties.
TEZ-2757. Fix download links for Tez releases.
TEZ-2742. VertexImpl.finished() terminationCause hides member var of the
same name
TEZ-2747. Update master to reflect 0.8.0-alpha release.
TEZ-2662. Provide a way to check whether AM or task opts are valid and error if not.
TEZ-2739. Improve handling of read errors in critical path analyzer
Release 0.8.0-alpha: 2015-09-01
INCOMPATIBLE CHANGES
TEZ-2048. Remove VertexManagerPluginContext.getTaskContainer()
TEZ-2565. Consider scanning unfinished tasks in VertexImpl::constructStatistics to reduce merge overhead.
TEZ-2468. Change the minimum Java version to Java 7.
ALL CHANGES:
TEZ-2749. TaskInfo in history parser should not depend on the apache directory project. Fix master build against hadoop-2.4
TEZ-2748. Fix master build against hadoop-2.2.
TEZ-2743. Fix TezContainerLauncher logging tokens.
TEZ-2708. Rename classes and variables post TEZ-2003 changes.
TEZ-2740. Create a reconfigureVertex alias for deprecated
setVertexParallelism API
TEZ-2690. Add critical path analyser
TEZ-2734. Add a test to verify the filename generated by OnDiskMerge.
TEZ-2732. DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers
TEZ-2687. ATS History shutdown happens before the min-held containers are released
TEZ-2629. LimitExceededException in Tez client when DAG has exceeds the default max counters
TEZ-2730. tez-api missing dependency on org.codehaus.jettison for json.
TEZ-2719. Consider reducing logs in unordered fetcher with shared-fetch option
TEZ-2646. Add scheduling casual dependency for attempts
TEZ-2647. Add input causality dependency for attempts
TEZ-2633. Allow VertexManagerPlugins to receive and report based on attempts
instead of tasks
TEZ-2650. Timing details on Vertex state changes
TEZ-2699. Internalize strings in ATF parser
TEZ-2701. Add time at which container was allocated to attempt
TEZ-2683. TestHttpConnection::testAsyncHttpConnectionInterrupt fails in certain environments.
TEZ-2692. bugfixes & enhancements related to job parser and analyzer.
TEZ-2663. SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM.
TEZ-2630. TezChild receives IP address instead of FQDN.
TEZ-2684. ShuffleVertexManager.parsePartitionStats throws IllegalStateException: Stats should be initialized.
TEZ-2172. FetcherOrderedGrouped using List to store InputAttemptIdentifier can lead to some inefficiency during remove() operation.
TEZ-2613. Fetcher(unordered) using List to store InputAttemptIdentifier can lead to some inefficiency during remove() operation.
TEZ-2645. Provide standard analyzers for job analysis.
TEZ-2627. Support for Tez Job Priorities.
TEZ-2623. Fix module dependencies related to hadoop-auth.
TEZ-2464. Move older releases to dist archive.
TEZ-2239. Update Tez UI docs to explain how to configure history url for YARN.
TEZ-2602. Throwing EOFException when launching MR job.
TEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition sizes from the source.
TEZ-2616. Fix build warning by undefined version of maven-findbugs-plugin.
TEZ-2588. Improper argument name
TEZ-2575. Handle KeyValue pairs size which do not fit in a single block.
TEZ-2599. Dont send obsoleted data movement events to tasks
TEZ-2542. TezDAGID fromString array length check.
TEZ-2565. Consider scanning unfinished tasks in VertexImpl::constructStatistics to reduce merge overhead.
TEZ-2296. Add option to print counters for tez-examples.
TEZ-2570. Fix license header issue for eps image files.
TEZ-2378. In case Fetcher (unordered) fails to do local fetch, log in debug mode to reduce log size.
TEZ-2558. Upload additional Tez images.
TEZ-2486. Update tez website to include links based on
http://www.apache.org/foundation/marks/pmcs.html#navigation.
TEZ-2548. TezClient submitDAG can hang if the AM is in the process of shutting down.
TEZ-2473. Consider using RawLocalFileSystem in MapOutput.createDiskMapOutput.
TEZ-2538. ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partitions.
TEZ-2489. Disable warn log for Timeline ACL error when tez.allow.disabled.timeline-domains set to true.
TEZ-2376. Remove TaskAttemptEventType.TA_DIAGNOSTICS_UPDATE
TEZ-2509. YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.
TEZ-2506. TestAysncHttpConnection failing.
TEZ-2503. findbugs version isn't reported properly in test-patch report.
TEZ-2198. Fix sorter spill counts.
TEZ-1883. Change findbugs version to 3.x.
TEZ-2440. Sorter should check for indexCacheList.size() in flush().
TEZ-2490. TEZ-2450 breaks Hadoop 2.2 and 2.4 compatability.
TEZ-2450. support async http clients in ordered & unordered inputs.
TEZ-2454. Change FetcherOrderedGroup to work as Callables instead of blocking threads.
TEZ-2466. tez-history-parser breaks hadoop 2.2 compatability.
TEZ-2463. Update site for 0.7.0 release
TEZ-2461. tez-history-parser compile fails with hadoop-2.4.
TEZ-2076. Tez framework to extract/analyze data stored in ATS for specific dag.
TEZ-2436. Tez UI: Add cancel button in column selector.
TEZ-2351. Remove GroupByOrderbyMRRTest example from tez-tests.
TEZ-2419. Inputs/Outputs should inform the Processor about Interrupts when interrupted during a blocking Op.
TEZ-1752. Inputs / Outputs in the Runtime library should be interruptable.
TEZ-1970. Fix javadoc warnings in SortMergeJoinExample.
TEZ-2003: Support for External services CHANGES
TEZ-2019. Temporarily allow the scheduler and launcher to be specified via configuration.
TEZ-2006. Task communication plane needs to be pluggable.
TEZ-2090. Add tests for jobs running in external services.
TEZ-2117. Add a manager for ContainerLaunchers running in the AM.
TEZ-2122. Setup pluggable components at AM/Vertex level.
TEZ-2123. Fix component managers to use pluggable components. (Enable hybrid mode)
TEZ-2125. Create a task communicator for local mode. Allow tasks to run in the AM.
TEZ-2131. Add additional tests for tasks running in the AM.
TEZ-2138. Fix minor bugs in adding default scheduler, getting launchers.
TEZ-2139. Update tez version to 0.7.0-TEZ-2003-SNAPSHOT.
TEZ-2175. Task priority should be available to the TaskCommunicator plugin.
TEZ-2187. Allow TaskCommunicators to report failed / killed attempts.
TEZ-2241. Miscellaneous fixes after last reabse.
TEZ-2283. Fixes after rebase 04/07.
TEZ-2284. Separate TaskReporter into an interface.
TEZ-2285. Allow TaskCommunicators to indicate task/container liveness.
TEZ-2302. Allow TaskCommunicators to subscribe for Vertex updates.
TEZ-2347. Expose additional information in TaskCommunicatorContext.
TEZ-2361. Propagate dag completion to TaskCommunicator.
TEZ-2381. Fixes after rebase 04/28.
TEZ-2388. Send dag identifier as part of the fetcher request string.
TEZ-2414. LogicalIOProcessorRuntimeTask, RuntimeTask, TezTaskRunner should handle interrupts & carry out necessary cleanups.
TEZ-2420. TaskRunner returning before executing the task.
TEZ-2433. Fixes after rebase 05/08
TEZ-2438. tez-tools version in the branch is incorrect.
TEZ-2434. Allow tasks to be killed in the Runtime.
TEZ-2443. TaskRunner2 should call abort, NPEs while cleaning up tasks.
TEZ-2465. Retrun the status of a kill request in TaskRunner2.
TEZ-2471. NPE in LogicalIOProcessorRuntimeTask while printing thread info.
TEZ-2495. Inform TaskCommunicaor about Task and Container termination reasons.
TEZ-2502. Fix for TezTaskRunner2 not killing tasks properly in all situations.
TEZ-2508. rebase 06/01
TEZ-2526. Fix version for tez-history-parser.
TEZ-2621. rebase 07/14
TEZ-2124. Change Node tracking to work per external container source.
TEZ-2004. Define basic interface for pluggable ContainerLaunchers.
TEZ-2005. Define basic interface for pluggable TaskScheduler.
TEZ-2651. Pluggable services should not extend AbstractService.
TEZ-2652. Cleanup the way services are specified for an AM and vertices.
TEZ-2653. Change service contexts to expose a user specified payload instead of the AM configuration.
TEZ-2441. Add tests for TezTaskRunner2.
TEZ-2657. Add tests for client side changes - specifying plugins, etc.
TEZ-2626. Fix log lines with DEBUG in messages, consolidate TEZ-2003 TODOs.
TEZ-2126. Add unit tests for verifying multiple schedulers, launchers, communicators.
TEZ-2698. rebase 08/05
TEZ-2675. Add javadocs for new pluggable components, fix problems reported by jenkins
TEZ-2678. Fix comments from reviews - part 1.
TEZ-2707. Fix comments from reviews - part 2.
TEZ-2713. Add tests for node handling when there's multiple schedulers.
TEZ-2721. rebase 08/14
TEZ-2714. Fix comments from review - part 3.
TEZ-2727. Fix findbugs warnings
TEZ-2670. Remove TaskAttempt holder used within TezTaskCommunicator.
TEZ-2735. rebase 08/21
TEZ-2736. Pre-merge: Update CHANGES.txt and version in branch.
Release 0.7.1: Unreleased
INCOMPATIBLE CHANGES
TEZ-2679. Admin forms of launch env settings
TEZ-2949. Allow duplicate dag names within session for Tez.
ALL CHANGES:
TEZ-3189. Pre-warm dags should not be counted in submitted dags count by DAGAppMaster.
TEZ-2967. Vertex start time should be that of first task start time in UI
TEZ-3175. Add tez client submit host
TEZ-3166. Fix a few cases where counters aren't fully updated and sent for failed tasks.
TEZ-2958. Recovered TA, whose commit cannot be recovered, should move to killed state
TEZ-3105. TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures.
TEZ-2863. Container, node, and logs not available in UI for tasks that fail to launch
TEZ-3140. Reduce AM memory usage during serialization
TEZ-3156. Tez client keeps trying to talk to RM even if RM does not know about the application.
TEZ-3115. Shuffle string handling adds significant memory overhead
TEZ-3149. Tez-tools: Add username in DagInfo.
TEZ-2988. DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier.
TEZ-3141. mapreduce.task.timeout is not translated to container heartbeat timeout
TEZ-3129. Tez task and task attempt UI needs application fails with NotFoundException
TEZ-3114. Shuffle OOM due to EventMetaData flood
TEZ-3102. Fetch failure of a speculated task causes job hang
TEZ-3126. Log reason for not reducing parallelism
TEZ-3123. Containers can get re-used even with conflicting local resources.
TEZ-3117. Deadlock in Edge and Vertex code
TEZ-3103. Shuffle can hang when memory to memory merging enabled
TEZ-3107. tez-tools: Log warn msgs in case ATS has wrong values (e.g startTime > finishTime).
TEZ-3104. Tez fails on Bzip2 intermediate output format on hadoop 2.7.1 and earlier
TEZ-3093. CriticalPathAnalyzer should be accessible via zeppelin.
TEZ-3089. TaskConcurrencyAnalyzer can return negative task count with very large jobs.
TEZ-2307. Possible wrong error message when submitting new dag.
TEZ-3076. Reduce merge memory overhead to support large number of in-memory mapoutputs
TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services.
TEZ-3036. Tez AM can hang on startup with no indication of error
TEZ-3052. Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED
TEZ-2937. Can Processor.close() be called after closing inputs and outputs?
TEZ-3037. History URL should be set regardless of which history logging service is enabled.
TEZ-3032. DAG start time getting logged using system time instead of recorded time in startTime field.
TEZ-2129. Task and Attempt views should contain links to the logs
TEZ-3025. InputInitializer creation should use the dag ugi.
TEZ-3017. HistoryACLManager does not have a close method for cleanup
TEZ-2914. Ability to limit vertex concurrency
TEZ-2918. Make progress notifications in IOs
TEZ-2952. NPE in TestOnFileUnorderedKVOutput
TEZ-808. Handle task attempts that are not making progress
TEZ-2987. TestVertexImpl.testTez2684 fails
TEZ-2599. Dont send obsoleted data movement events to tasks
TEZ-2943. Change shuffle vertex manager to use per vertex data for auto
TEZ-2633. Allow VertexManagerPlugins to receive and report based on attempts
TEZ-3011. Link Vertex Name in Dag Tasks/Task Attempts to Vertex
TEZ-2538. ADDITIONAL_SPILL_COUNT wrongly populated for DefaultSorter with multiple partitions.
TEZ-3006. Remove unused import in TestHistoryParser.
TEZ-2979. FlakyTest: org.apache.tez.history.TestHistoryParser.
TEZ-2684. ShuffleVertexManager.parsePartitionStats throws IllegalStateException: Stats should be initialized.
TEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition sizes from the source.
TEZ-2995. Timeline primary filter should only be on callerId and not type.
TEZ-2824. Add javadocs for Vertex.setConf and DAG.setConf.
TEZ-2963. RecoveryService#handleSummaryEvent exception with HDFS transparent encryption + kerberos authentication.
TEZ-2966. Tez does not honor mapreduce.task.timeout
TEZ-2346. TEZ-UI: Lazy load other info / counter data
TEZ-2975. Bump up apache commons dependency.
TEZ-2970. Re-localization in TezChild does not use correct UGI.
TEZ-2968. Counter limits exception causes AM to crash.
TEZ-2947. Tez UI: Timeline, RM & AM requests gets into a consecutive loop in counters page without any delay
TEZ-2949. Allow duplicate dag names within session for Tez.
TEZ-2923. Tez Live UI counters view empty for vertices, tasks, attempts
TEZ-2935. Add MR slow start translation for ShuffleVertexManager
TEZ-2940. Invalid shuffle max slow start setting causes vertex to hang indefinitely
TEZ-1670. Add tests for all converter functions in HistoryEventTimelineConversion.
TEZ-2922. Tez Live UI gives access denied for admins
TEZ-2828. Fix typo in "Shuffle assigned " log statement in shuffle.orderedgrouped.Shuffle.
TEZ-2900. Ignore V_INPUT_DATA_INFORMATION when vertex is in Failed/Killed/Error
TEZ-2904. Pig can't specify task specific command opts
TEZ-2899. Tez UI: DAG getting created with huge horizontal gap in between vertices
TEZ-2882. Consider improving fetch failure handling
TEZ-2907. NPE in IFile.Reader.getLength during final merge operation
TEZ-2850. Tez MergeManager OOM for small Map Outputs
TEZ-2886. Ability to merge AM credentials with DAG credentials.
TEZ-2896. Fix thread names used during Input/Output initialization.
TEZ-2866. Tez UI: Newly added columns wont be displayed by default in tables
TEZ-2885. Remove counter logs from AMWebController.
TEZ-2887. Tez build failure due to missing dependency in pom files.
TEZ-2096. TEZ-UI : Add link to view AM log of finished & running apps
TEZ-2874. Improved logging for caller context.
TEZ-2781. Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails
TEZ-2868. Fix setting Caller Context in Tez Examples.
TEZ-2860. NPE in DAGClientImpl.
TEZ-2855. Fix a potential NPE while routing VertexManager events.
TEZ-2758. Remove append API in RecoveryService after TEZ-1909.
TEZ-2851. Support a way for upstream applications to pass in a caller context to Tez.
TEZ-2858. Stop using System.currentTimeMillis in TestInputReadyTracker.
TEZ-2857. Fix flakey tests in TestDAGImpl.
TEZ-2398. Flaky test: TestFaultTolerance
TEZ-2808. Race condition between preemption and container assignment
TEZ-2853. Tez UI: task attempt page is coming empty
TEZ-2716. DefaultSorter.isRleNeeded not thread safe
TEZ-2847. Tez UI: Task details doesn't gets updated on manual refresh after job complete
TEZ-2843. Tez UI: Show error if in progress fails due to AM not reachable
TEZ-2842. Tez UI: Update Tez App details page while in-progress
TEZ-2834. Make Tez preemption resilient to incorrect free resource reported
by YARN
TEZ-2775. Improve and consolidate logging in Runtime components.
TEZ-2097. TEZ-UI Add dag logs backend support
TEZ-2812. Preemption sometimes does not respect heartbeats between preemptions
TEZ-814. Improve heuristic for determining a task has failed outputs
TEZ-2829. Tez UI: minor fixes to in-progress update of UI from AM
TEZ-2663. SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM.
TEZ-2825. Report progress in terms of completed tasks to reduce load on AM for Tez UI
TEZ-2812. Tez UI: Update task & attempt tables while in progress.
TEZ-2786. Tez UI: Update vertex, task & attempt details page while in progress.
TEZ-2817. Tez UI: update in progress counter data for the dag vertices and tasks table
TEZ-2813. Tez UI: add counter data for rest api calls to AM Web Services v2
TEZ-2660. Tez UI: need to show application page even if system metrics publish is disabled.
TEZ-2787. Tez AM should have java.io.tmpdir=./tmp to be consistent with tasks
TEZ-2780. Tez UI: Update All Tasks page while in progress
TEZ-2792. Add AM web service API for tasks
TEZ-2807. Log data in the finish event instead of the start event
TEZ-2766. Tez UI: Add vertex in-progress info in DAG details
TEZ-2768. Log a useful error message when the summary stream cannot be closed when shutting
down an AM.
TEZ-2745. ClassNotFoundException of user code should fail dag
TEZ-2761. Tez UI: update the progress on the dag and vertices pages with info from AM
TEZ-2731. Fix Tez GenericCounter performance bottleneck
TEZ-2752. logUnsuccessful completion in Attempt should write original finish
time to ATS
TEZ-2755. Fix findbugs warning in TezClient
TEZ-2767. Make TezMxBeanResourceCalculator the default resource calculator.
TEZ-2602. Throwing EOFException when launching MR job
TEZ-2575. Handle KeyValue pairs size which do not fit in a single block in PipelinedSorter
TEZ-2198. Fix sorter spill counts
TEZ-2440. Sorter should check for indexCacheList.size() in flush()
TEZ-2742. VertexImpl.finished() terminationCause hides member var of the
same name
TEZ-2662. Provide a way to check whether AM or task opts are valid and error if not.
TEZ-2300. TezClient.stop() takes a lot of time or does not work sometimes
TEZ-2734. Add a test to verify the filename generated by OnDiskMerge.
TEZ-2732. DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers
TEZ-2687. ATS History shutdown happens before the min-held containers are released
TEZ-2629. LimitExceededException in Tez client when DAG has exceeds the default max counters
TEZ-2540. Create both tez-dist minimal and minimal.tar.gz formats as part of build
TEZ-2630. TezChild receives IP address instead of FQDN.
TEZ-2211. Tez UI: Allow users to configure timezone
TEZ-2623. Fix module dependencies related to hadoop-auth.
TEZ-1314. Port MAPREDUCE-5821 to Tez.
TEZ-2568. V_INPUT_DATA_INFORMATION may happen after vertex is initialized
TEZ-2291. TEZ UI: Improper vertex name in tables.
TEZ-2567. Tez UI: download dag data does not work within ambari
TEZ-2559. tez-ui fails compilation due to version dependency of frontend-maven-plugin
TEZ-2545. It is not necessary to start the vertex group commit when DAG is in TERMINATING
TEZ-2554. Tez UI: View log link does not correctly propagate login crendential to read log from yarn web.
TEZ-2548. TezClient submitDAG can hang if the AM is in the process of shutting down.
TEZ-2547. Tez UI: Download Data fails on secure, cross-origin clusters
TEZ-1961. Remove misleading exception "No running dag" from AM logs.
TEZ-2546. Tez UI: Fetch hive query text from timeline if dagInfo is not set.
TEZ-2513. Tez UI: Allow filtering by DAG ID on All dags table.
TEZ-2541. DAGClientImpl enable TimelineClient check is wrong.
TEZ-2539. Tez UI: Pages are not updating in IE.
TEZ-2535. Tez UI: Failed task attempts link in vertex details page is broken.
TEZ-2489. Disable warn log for Timeline ACL error when tez.allow.disabled.timeline-domains set to true.
TEZ-2528. Tez UI: Column selector buttons gets clipped, and table scroll bar not visible in mac.
TEZ-2391. TestVertexImpl timing out at times on jenkins builds.
TEZ-2509. YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.
TEZ-2527. Tez UI: Application hangs on entering erroneous RegEx in counter table search box
TEZ-2523. Tez UI: derive applicationId from dag/vertex id instead of relying on json data
TEZ-2505. PipelinedSorter uses Comparator objects concurrently from multiple threads.
TEZ-2504. Tez UI: tables - show status column without scrolling, numeric 0 shown as Not available
TEZ-2478. Move OneToOne routing to store events in Tasks.
TEZ-2482. Tez UI: Mouse events not working on IE11
TEZ-1529. ATS and TezClient integration in secure kerberos enabled cluster.
TEZ-2481. Tez UI: graphical view does not render properly on IE11
TEZ-2474. The old taskNum is logged incorrectly when parallelism is changed
TEZ-2460. Temporary solution for issue due to YARN-2560
TEZ-2455. Tez UI: Dag view caching, error handling and minor layout changes
TEZ-2453. Tez UI: show the dagInfo is the application has set the same.
TEZ-2447. Tez UI: Generic changes based on feedbacks.
TEZ-2409. Allow different edges to have different routing plugins
Release 0.7.0: 2015-05-18
INCOMPATIBLE CHANGES
TEZ-2176. Move all logging to slf4j. (commons-logging jar no longer part of Tez tar)
TEZ-1993. Implement a pluggable InputSizeEstimator for grouping fairly.
TEZ-2424. Bump up max counter group name length limit to account for per_io counters.
Default max limit increased. Should not affect existing users.
ALL CHANGES:
TEZ-2446. Tez UI: Add tezVersion details when downloading timeline data for offline use
TEZ-2432. Syntax error in DOAP file release section
TEZ-2445. Disable the object cleanup in local mode in LogicalIOProcessorRuntimeTask.
TEZ-2057. tez-dag/pom.xml contains versions for dependencies.
TEZ-2435. Add public key to KEYS
TEZ-2421. Deadlock in AM because attempt and vertex locking each other out
TEZ-2426. Ensure the eventRouter thread completes before switching to a new task and thread safety fixes in IPOContexts.
TEZ-2412. Should kill vertex in DAGImpl#VertexRerunWhileCommitting
TEZ-2410. VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
TEZ-776. Reduce AM mem usage caused by storing TezEvents
TEZ-2423. Tez UI: Remove Attempt Index column from task->attempts page
TEZ-2416. Tez UI: Make tooltips display faster.
TEZ-2404. Handle DataMovementEvent before its TaskAttemptCompletedEvent
TEZ-2424. Bump up max counter group name length limit to account for per_io counters.
TEZ-2417. Tez UI: Counters are blank in the Attempts page if all attempts failed
TEZ-2366. Pig tez MiniTezCluster unit tests fail intermittently after TEZ-2333
TEZ-2406. Tez UI: Display per-io counter columns in task and attempt pages under vertex
TEZ-2384. Add warning message in the case of prewarn under non-session mode.
TEZ-2415. PMC RDF needs to use asfext:pmc, not asfext:PMC.
TEZ-2401. Tez UI: All-dag page has duration keep counting for KILLED dag.
TEZ-2392. Have all readers throw an Exception on incorrect next() usage.
TEZ-2408. TestTaskAttempt fails to compile against hadoop-2.4 and hadoop-2.2.
TEZ-2405. PipelinedSorter can throw NPE with custom compartor.
TEZ-1897. Create a concurrent version of AsyncDispatcher
TEZ-2394. Issues when there is an error in VertexManager callbacks
TEZ-2386. Tez UI: Inconsistent usage of icon colors
TEZ-2395. Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.
TEZ-2360. per-io counters flag should generate both overall and per-edge counters
TEZ-2389. Tez UI: Sort by attempt-no is incorrect in attempts pages.
TEZ-2363: Fix off-by-one error in REDUCE_INPUT_RECORDS counter
TEZ-2383. Cleanup input/output/processor contexts in LogicalIOProcessorRuntimeTask.
TEZ-2084. Tez UI: Stacktrace format info is lost in diagnostics
TEZ-2374. Fix build break against hadoop-2.2 due to TEZ-2325.
TEZ-2314. Tez task attempt failures due to bad event serialization
TEZ-2368. Make a dag identifier available in Context classes.
TEZ-2325. Route status update event directly to the attempt.
TEZ-2358. Pipelined Shuffle: MergeManager assumptions about 1 merge per source-task.
TEZ-2342. TestFaultTolerance.testRandomFailingTasks fails due to timeout.
TEZ-2362. State Change Notifier Thread should be stopped when dag is
completed
TEZ-2364. Resolve config parameter replacement on the client, before sending them to the AM.
TEZ-2298. Avoid logging full exception trace in TaskRunner when it's not the main error reason and is ignored.
TEZ-2248. VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
TEZ-2341. TestMockDAGAppMaster.testBasicCounters fails on windows
TEZ-2352. Move getTaskStatistics into the RuntimeTask class.
TEZ-2357. Tez UI: misc.js.orig is committed by accident
TEZ-2261. Should add diagnostics in DAGAppMaster when recovery error happens
TEZ-2340. TestRecoveryParser fails
TEZ-2345. Tez UI: Enable cell level loading in all DAGs table
TEZ-2330. Create reconfigureVertex() API for input based initialization
TEZ-2292. Add e2e test for error reporting when vertex manager invokes
plugin APIs
TEZ-2308. Add set/get of record counts in task/vertex statistics
TEZ-2344. Tez UI: Equip basic-ember-table's cell level loading for all use cases in all DAGs table
TEZ-2313. Regression in handling obsolete events in ShuffleScheduler.
TEZ-2212. Notify components on DAG completion.
TEZ-2328. Add tez.runtime.sorter.class & rename tez.runtime.sort.threads
to tez.runtime.pipelined.sorter.sort.threads.
TEZ-2333. Enable local fetch optimization by default.
TEZ-2310. Deadlock caused by StateChangeNotifier sending notifications on
thread holding locks
TEZ-1969. Stop the DAGAppMaster when a local mode client is stopped
TEZ-714. OutputCommitters should not run in the main AM dispatcher thread
TEZ-2323. Fix TestOrderedWordcount to use MR memory configs.
TEZ-1482. Fix memory issues for Local Mode running concurrent tasks
TEZ-2033. Update TestOrderedWordCount to add processor configs as history text
and use MR configs correctly
TEZ-2318. Tez UI: source and sink page is broken as they are not populated.
TEZ-2016. Tez UI: Dag View Fit and Finish
TEZ-2252. Tez UI: in graphical view some of the sinks are hidden as they overlap
TEZ-2275. Tez UI: enable faster loading and caching of data in tables
TEZ-2234. Add API for statistics information - allow vertex managers to get
output size per source vertex
TEZ-2274. Tez UI: full data loading, client side search and sort for other pages
TEZ-2301. Switch Tez Pre-commit builds to use tezqa user.
TEZ-2299. Invalid dag creation in MRRSleepJob post TEZ-2293.
TEZ-2290. Scale memory for Default Sorter down to a max of 2047 MB if configured higher.
TEZ-2233. Allow EdgeProperty of an edge to be changed by VertexManager
TEZ-2293. When running in "mr" mode, always use MR config settings.
TEZ-2273. Tez UI: Support client side searching & sorting for dag tasks page
TEZ-2223. TestMockDAGAppMaster fails due to TEZ-2210 on mac.
TEZ-2236. Tez UI: Support loading of all tasks in the dag tasks page
TEZ-2159. Tez UI: download timeline data for offline use.
TEZ-2269. DAGAppMaster becomes unresponsive (post TEZ-2149).
TEZ-2243. documentation should explicitly specify protobuf 2.5.0.
TEZ-2232. Allow setParallelism to be called multiple times before tasks get
scheduled
TEZ-2265. All inputs/outputs in a task share the same counter object
TEZ-2251. Race condition in VertexImpl & Edge causes DAG to hang.
TEZ-2264. Remove unused taskUmbilical reference in TezTaskRunner, register as running late.
TEZ-2149. Optimizations for the timed version of DAGClient.getStatus.
TEZ-2213. For the ordered case, enabling pipelined shuffle should automatically disable final merge.
TEZ-2204. TestAMRecovery increasingly flaky on jenkins builds.
TEZ-2209. Fix pipelined shuffle to fetch data from any one attempt
TEZ-2210. Record DAG AM CPU usage stats
TEZ-2203. Intern strings in tez counters
TEZ-2202. Fix LocalTaskExecutionThread ID to the standard thread numbering.
TEZ-2059. Remove TaskEventHandler in TestDAGImpl
TEZ-2191. Simulation improvements to MockDAGAppMaster
TEZ-2195. TestTezJobs::testInvalidQueueSubmission/testInvalidQueueSubmissionToSession
fail with hadoop branch-2.
TEZ-1827. MiniTezCluster takes 10 minutes to shut down.
TEZ-2178. YARN-3122 breaks tez compilation with hadoop 2.7.0.
TEZ-2174. Make task priority available to TaskAttemptListener.
TEZ-2169. Add NDC context to various threads and pools.
TEZ-2171. Remove unused metrics code.
TEZ-2001. Support pipelined data transfer for ordered output.
TEZ-2170. Incorrect its in README.md.
TEZ-2070. Controller class of output should be committer rather than initializer in DAG's dot file.
TEZ-2083. Make PipelinedSorter as the default sorter.
TEZ-1967. Add a monitoring API on DAGClient which returns after a time interval or on DAG final state change.
TEZ-2130. Send the sessionToken as part of the AM CLC.
TEZ-1935. Organization should be removed from http://tez.apache.org/team-list.html.
TEZ-2009. Change license/copyright headers to 2015.
TEZ-2085. PipelinedSorter should bail out (on BufferOverflowException) instead of retrying continuously.
TEZ-167. Create tests for MR Combiner.
TEZ-2080. Localclient should be using tezconf in init instead of yarnconf.
TEZ-2072. Add missing Private annotation to createDAG in the DAG API class.
TEZ-2095. master branch fails to compile against hadoop-2.4.
TEZ-2093. Add events to MockDAGAppMaster and add e2e test for event routing
TEZ-2075. Incompatible issue caused by TEZ-1233 that TezConfiguration.TEZ_SITE_XML is made private
TEZ-2082. Race condition in TaskAttemptListenerImpTezDag.getTask()
TEZ-1233. Allow configuration of framework parameters per vertex.
TEZ-2045. TaskAttemptListener should not pull Tasks from AMContainer. Instead these should be registered with the listener.
TEZ-1914. VertexManager logic should not run on the central dispatcher
TEZ-2023. Refactor logIndividualFetchComplete() to be common for both shuffle-schedulers.
TEZ-1999. IndexOutOfBoundsException during merge.
TEZ-2000. Source vertex exists error during DAG submission.
TEZ-2008. Add methods to SecureShuffleUtils to verify a reply based on a provided Key.
TEZ-1995. Build failure against hadoop 2.2.
TEZ-1997. Remove synchronization DefaultSorter::isRLENeeded() (Causes sorter to hang indefinitely in large jobs).
TEZ-1996. Update Website after 0.6.0
TEZ-1803. Support > 2gb sort buffer in pipelinedsorter.
TEZ-1826. Add option to disable split grouping and local mode option for tez-examples.
TEZ-1982. TezChild setupUgi should not be using environment.
TEZ-1980. Suppress tez-dag findbugs warnings until addressed.
TEZ-1855. Avoid scanning for previously written files within Inputs / Outputs.
TEZ-1902. Fix findbugs warnings in tez-mapreduce.
TEZ-1963. Fix post memory merge to be > 2 GB.
TEZ-1901. Fix findbugs warnings in tez-examples.
TEZ-1941. Memory provided by *Context.getAvailableMemory needs to be setup explicitly.
TEZ-1879. Create local UGI instances for each task and the AM, when running in LocalMode.
TEZ-1661. LocalTaskScheduler hangs when shutdown.
TEZ-1951. Fix general findbugs warnings in tez-dag.
TEZ-1905. Fix findbugs warnings in tez-tests.
TEZ-1945. Remove 2 GB memlimit restriction in MergeManager.
TEZ-1913. Reduce deserialize cost in ValuesIterator.
TEZ-1917. Examples should extend TezExampleBase.
TEZ-1892. Add hashCode and equals for Vertex/VertexGroup/Edge/GroupInputEdge.
TEZ-1904. Fix findbugs warnings in tez-runtime-library module.
TEZ-1903. Fix findbugs warnings in tez-runtime-internal module.
TEZ-1896. Move the default heartbeat timeout and checkinterval to TezConfiguration.
TEZ-1274. Remove Key/Value type checks in IFile.
TEZ-1912. Merge exceptions are thrown when enabling tez.runtime.shuffle.memory-to-memory.enable && tez.runtime.shuffle.memory-to-memory.segments.
TEZ-1922. Fix comments: add UNSORTED_OUTPUT to TEZ_TASK_SCALE_MEMORY_WEIGHTED_RATIOS.
TEZ-485. Get rid of TezTaskStatus.
TEZ-1899. Fix findbugs warnings in tez-common module.
TEZ-1898. Fix findbugs warnings in tez-api module.
TEZ-1906. Fix findbugs warnings in tez-yarn-timeline-history-with-acls.
TEZ-1767. Enable RLE in reducer side merge codepath.
TEZ-1837. Restrict usage of Environment variables to main methods.
TEZ-1867. Create new central dispatcher for Tez AM
TEZ-1844. Shouldn't invoke system.exit in local mode when AM is failed to start.
TEZ-1889. Fix test-patch to provide correct findbugs report.
TEZ-1313. Setup pre-commit build to test submitted patches.
TEZ-1856. Remove LocalOnFileSortedOutput, LocalMergedInput, LocalTaskOutputFiles.
TEZ-1949. Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges.
TEZ-1593. Refactor PipelinedSorter to remove all MMAP based ByteBuffer references.
Release 0.6.3: Unreleased
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-2907. NPE in IFile.Reader.getLength during final merge operation
TEZ-2850. Tez MergeManager OOM for small Map Outputs
TEZ-2781. Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails
TEZ-2855. Fix a potential NPE while routing VertexManager events.
TEZ-2716. DefaultSorter.isRleNeeded not thread safe
TEZ-2758. Remove append API in RecoveryService after TEZ-1909.
TEZ-2851. Support a way for upstream applications to pass in a caller context to Tez.
TEZ-2398. Flaky test: TestFaultTolerance
TEZ-2808. Race condition between preemption and container assignment
TEZ-2834. Make Tez preemption resilient to incorrect free resource reported
by YARN
TEZ-2097. TEZ-UI Add dag logs backend support
TEZ-2812. Preemption sometimes does not respect heartbeats between preemptions
TEZ-814. Improve heuristic for determining a task has failed outputs
TEZ-2809. Minimal distribution compiled on 2.6 fails to run on 2.7
TEZ-2768. Log a useful error message when the summary stream cannot be closed when shutting
down an AM.
TEZ-2745. ClassNotFoundException of user code should fail dag
TEZ-2752. logUnsuccessful completion in Attempt should write original finish
time to ATS
TEZ-2742. VertexImpl.finished() terminationCause hides member var of the
same name
TEZ-2732. DefaultSorter throws ArrayIndex exceptions on 2047 Mb size sort buffers
TEZ-2290. Scale memory for Default Sorter down to a max of 2047 MB if configured higher.
TEZ-2734. Add a test to verify the filename generated by OnDiskMerge.
TEZ-2687. ATS History shutdown happens before the min-held containers are released
TEZ-2629. LimitExceededException in Tez client when DAG has exceeds the default max counters
TEZ-2630. TezChild receives IP address instead of FQDN.
Release 0.6.2: 2015-08-07
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-2311. AM can hang if kill received while recovering from previous attempt.
TEZ-2623. Fix module dependencies related to hadoop-auth.
TEZ-2560. fix tex-ui build for maven 3.3+
TEZ-2600. When used with HDFS federation(viewfs) ,tez will throw a error
TEZ-2579. Incorrect comparison of TaskAttemptId
TEZ-2549. Reduce Counter Load on the Timeline Server
TEZ-2548. TezClient submitDAG can hang if the AM is in the process of shutting down.
TEZ-2534. Error handling summary event when shutting down AM.
TEZ-2511. Add exitCode to diagnostics when container fails.
TEZ-2489. Disable warn log for Timeline ACL error when tez.allow.disabled.timeline-domains set to true.
TEZ-2509. YarnTaskSchedulerService should not try to allocate containers if AM is shutting down.
TEZ-2483. Tez should close task if processor fail
Release 0.6.1: 2015-05-18
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-2437. FilterLinesByWord NPEs when run in Localmode
TEZ-2057. tez-dag/pom.xml contains versions for dependencies.
TEZ-2282. Delimit reused yarn container logs (stderr, stdout, syslog) with task attempt start/stop events
TEZ-2396. pig-tez-tfile-parser pom is hard coded to depend on 0.6.0-SNAPSHOT version.
TEZ-2237. Valid events should be sent out when an Output is not started.
TEZ-1988. Tez UI: does not work when using file:// in a browser
TEZ-2390. tez-tools swimlane tool fails to parse large jobs >8K containers
TEZ-2256. Avoid use of BufferTooSmallException to signal end of buffer in UnorderedPartitionedKVWriter
TEZ-2380. Disable fall back to reading from timeline if timeline disabled.
TEZ-2226. Disable writing history to timeline if domain creation fails.
TEZ-2259. Push additional data to Timeline for Recovery for better consumption in UI.
TEZ-2365. Update tez-ui war's license/notice to reflect OFL license correctly.
TEZ-2329. UI Query on final dag status performance improvement
TEZ-2287. Deprecate VertexManagerPluginContext.getTaskContainer().
TEZ-1909. Remove need to copy over all events from attempt 1 to attempt 2 dir
TEZ-2061. Tez UI: vertex id column and filter on tasks page should be changed to vertex name
TEZ-2242. Refactor ShuffleVertexManager code
TEZ-2205. Tez still tries to post to ATS when yarn.timeline-service.enabled=false.
TEZ-2047. Build fails against hadoop-2.2 post TEZ-2018
TEZ-2064. SessionNotRunning Exception not thrown is all cases
TEZ-2189. Tez UI live AM tracking url only works for localhost addresses
TEZ-2179. Timeline relatedentries missing cause exaggerated warning.
TEZ-2168. Fix application dependencies on mutually exclusive artifacts: tez-yarn-timeline-history
and tez-yarn-timeline-history-with-acls.
TEZ-2190. TestOrderedWordCount fails when generateSplitsInClient set to true.
TEZ-2091. Add support for hosting TEZ_UI with nodejs.
TEZ-2165. Tez UI: DAG shows running status if killed by RM in some cases.
TEZ-2158. TEZ UI: Display dag/vertex names, and task/attempt index in breadcrumb.
TEZ-2160. Tez UI: App tracking URL should support navigation back.
TEZ-2147. Swimlanes: Improved tooltip
TEZ-2142. TEZ UI: Breadcrumb border color looks out of place in wrapped mode.
TEZ-2134. TEZ UI: On request failure, display request URL and server name in error bar.
TEZ-2136. Some enhancements to the new Tez UI.
TEZ-2135. ACL checks handled incorrectly in AMWebController.
TEZ-1990. Tez UI: DAG details page shows Nan for end time when a DAG is running.
TEZ-2116. Tez UI: dags page filter does not work if more than one filter is specified.
TEZ-2106. TEZ UI: Display data load time, and add a refresh button for items that can be refreshed.
TEZ-2114. Tez UI: task/task attempt status is not available when its running.
TEZ-2112. Tez UI: fix offset calculation, add home button to breadcrumbs.
TEZ-2038. TEZ-UI DAG is always running in tez-ui when the app is failed but no DAGFinishedEvent is logged.
TEZ-2102. Tez UI: DAG view has hidden edges, dragging DAG by holding vertex causes unintended click.
TEZ-2101. Tez UI: Issues on displaying a table.
TEZ-2092. Tez UI history url handler injects spurious trailing slash.
TEZ-2098. Tez UI: Dag details should be the default page for dag, fix invalid time entries for failed Vertices.
TEZ-2024. TaskFinishedEvent may not be logged in recovery.
TEZ-2031. Tez UI: horizontal scrollbars do not appear in tables, causing them to look truncated.
TEZ-2073. SimpleHistoryLoggingService cannot be read by log aggregation (umask)
TEZ-2078. Tez UI: Task logs url use in-progress url causing various errors.
TEZ-2077. Tez UI: No diagnostics on Task Attempt Details page if task attempt failed.
TEZ-2065. Setting up tez.tez-ui.history-url.base with a trailing slash can result in failures to redirect correctly.
TEZ-2068. Tez UI: Dag view should use full window height, disable webuiservice in localmode.
TEZ-2079. Tez UI: trailing slash in timelineBaseUrl in ui should be handled.
TEZ-2069. Tez UI: appId should link to application in dag details view.
TEZ-2063. Tez UI: Flaky log url in tasks table.
TEZ-2062. Tez UI: Showing 50 elements not working properly.
TEZ-2056. Tez UI: fix VertexID filter,show only tez configs by default,fix appattemptid.
TEZ-2052. Tez UI: log view fixes, show version from build, better handling of ats url config.
TEZ-2043. Tez UI: add progress info from am webservice to dag and vertex views.
TEZ-2032. Update CHANGES.txt to show 0.6.0 is released
TEZ-2018. App Tracking and History URL should point to the Tez UI.
TEZ-2035. Make timeline server putDomain exceptions non-fatal - work-around
TEZ-1929. pre-empted tasks should be marked as killed instead of failed
TEZ-2017. TEZ UI - Dag view throwing error whild re-displaying additionals in some dags.
TEZ-2013. TEZ UI - App Details Page UI Nits
TEZ-2014. Tez UI: Nits : All tables, Vertices Page UI.
TEZ-2012. TEZ UI: Show page number in all tables, and display more readable task/attempt ids.
TEZ-1973. Dag View
TEZ-2010. History payload generated from conf has ${var} placeholders.
TEZ-1946. Tez UI: add source & sink views, add counters to vertices/all task views.
TEZ-1987. Tez UI non-standalone mode uses invalid protocol.
TEZ-1983. Tez UI swimlane task attempt link is broken
Release 0.6.0: 2015-01-23
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-1977. Fixup CHANGES.txt with Tez UI jiras
TEZ-1743. Add versions-maven-plugins artifacts to gitignore
TEZ-1968. Tez UI - All vertices of DAG are not listed in vertices page
TEZ-1890. tez-ui web.tar.gz also being uploaded to maven repository
TEZ-1938. Build warning duplicate jersey-json definitions
TEZ-1910. Build fails against hadoop-2.2.0.
TEZ-1882. Tez UI build does not work on Windows
TEZ-1915. Add public key to KEYS
TEZ-1907. Fix javadoc warnings in tez codebase
TEZ-1891. Incorrect number of Javadoc warnings reported
TEZ-1762. Lots of unit tests do not have timeout parameter set.
TEZ-1886. remove deprecation warnings for tez-ui on the console.
TEZ-1875. dropdown filters do not work on vertices and task attempts page.
TEZ-1873. TestTezAMRMClient fails due to host resolution timing out.
TEZ-1881. Setup initial test-patch script for TEZ-1313.
TEZ-1864. move initialization code dependent on config params to App.ready.
TEZ-1870. Time displayed in the UI is in GMT.
TEZ-1858. Docs for deploying/using the Tez UI.
TEZ-1859. TestGroupedSplits has commented out test: testGzip.
TEZ-1868. Document how to do Windows builds due to with ACL symlink build changes.
TEZ-1872. docs/src/site/custom/project-info-report.properties needs license header.
TEZ-1850. Enable deploy for tez-ui war.
TEZ-1841. Remove range versions for dependencies in tez-ui.
TEZ-1854. Failing tests due to host resolution timing out.
TEZ-1860. mvn apache-rat:check broken for tez-ui.
TEZ-1866. remove the "original" directory under tez-ui
TEZ-1591. Add multiDAG session test and move TestLocalMode to tez-tests
TEZ-1769. ContainerCompletedWhileRunningTransition should inherit from TerminatedWhileRunningTransition
TEZ-1849. Fix tez-ui war file licensing.
TEZ-1840. Document TezTaskOutput.
TEZ-1576. Class level comment in {{MiniTezCluster}} ends abruptly.
TEZ-1838. tez-ui/src/main/webapp/bower.json gets updated after compiling source code.
TEZ-1789. Move speculator processing off the central dispatcher.
TEZ-1610. Add additional task counters for fetchers, merger.
TEZ-1847. Fix package name for MiniTezClusterWithTimeline.
TEZ-1846. Build fails with package org.apache.tez.dag.history.logging.ats does not exist.
TEZ-1696. Make Tez use the domain-based timeline ACLs.
TEZ-1835. TestFaultTolerance#testRandomFailingTasks is timing out
TEZ-1832. TestSecureShuffle fails with NoClassDefFoundError: org/bouncycastle/x509/X509V1CertificateGenerator
TEZ-1672. Update jetty to use stable 7.x version - 7.6.16.v20140903.
TEZ-1822. Docs for Timeline/ACLs/HistoryText.
TEZ-1252. Change wording on http://tez.apache.org/team-list.html related to member confusion.
TEZ-1805. Tez client DAG cycle detection should detect self loops
TEZ-1816. It is possible to receive START event when DAG is failed
TEZ-1787. Counters for speculation
TEZ-1773. Add attempt failure cause enum to the attempt failed/killed
history record
TEZ-14. Support MR like speculation capabilities based on latency deviation
from the mean
TEZ-1733. TezMerger should sort FileChunks on size when merging
TEZ-1738. Tez tfile parser for log parsing
TEZ-1627. Remove OUTPUT_CONSUMABLE and related Event in TaskAttemptImpl
TEZ-1736. Add support for Inputs/Outputs in runtime-library to generate history text data.
TEZ-1721. Update INSTALL instructions for clarifying tez client jars
compatibility with runtime tarball on HDFS.
TEZ-1690. TestMultiMRInput tests fail because of user collisions.
TEZ-1687. Use logIdentifier of Vertex for logging.
TEZ-1737. Should add taskNum in VertexFinishedEvent.
TEZ-1772. Failing tests post TEZ-1737.
TEZ-1785. Remove unused snappy-java dependency.
TEZ-1685. Remove YARNMaster which is never used.
TEZ-1797. Create necessary content for Tez DOAP file.
TEZ-1650. Please create a DOAP file for your TLP.
TEZ-1697. DAG submission fails if a local resource added is already part of tez.lib.uris
TEZ-1060 Add randomness to fault tolerance tests
TEZ-1790. DeallocationTaskRequest may been handled before corresponding AllocationTaskRequest in local mode
TEZ-UI CHANGES (TEZ-8):
TEZ-1823. default ATS url should be the same host as ui
TEZ-1817. Add configuration and build details to README
TEZ-1813. display loading and other error messages in tez-ui
TEZ-1809. Provide a error bar to report errors to users in Tez-UI
TEZ-1810. Do not deploy tez-ui war to maven repo
TEZ-1709. Bunch of files in tez-ui missing Apache license header
TEZ-1784. Attempt details in tasks table.
TEZ-1801. Update build instructions for tez-ui
TEZ-1757. Column selector for tables.
TEZ-1794. Vertex view needs a task attempt rollup
TEZ-1791. breadcrumbs for moving between pages.
TEZ-1781. Configurations view ~ New design
TEZ-1605. Landing page for Tez UI
TEZ-1600. Swimlanes View for Tez UI
TEZ-1799. Enable Cross Origin Support in Tez UI
TEZ-1634. Fix compressed IFile shuffle errors
TEZ-1615. Skeleton framework for Tez UI
TEZ-1604. Task View for Tez UI
TEZ-1603. Vertex View for Tez UI.
TEZ-1720. Allow filters in all tables and also to pass in filters using url params.
TEZ-1708. Make UI part of TEZ build process.
TEZ-1617. Shim layer for Tez UI for use within Ambari.
TEZ-1741. App view.
TEZ-1751. Log view & download links in task and task attempt view.
TEZ-1753. Queue in dags view.
TEZ-1765. Allow dropdown lists in table filters.
TEZ-1606. Counters View for DAG, Vertex, and Task.
TEZ-1768. follow up jira to address minor issues in Tez-ui.
TEZ-1783. Wrapper in standalone mode.
TEZ-1820. Fix wrong links.
Release 0.5.5: Unreleased
INCOMPATIBLE CHANGES
TEZ-2552. CRC errors can cause job to run for very long time in large jobs.
ALL CHANGES:
TEZ-2781. Fallback to send only TaskAttemptFailedEvent if taskFailed heartbeat fails