private void addResourceRequest( Priority priority, String resourceName, Resource capability, T req, boolean relaxLocality, String labelExpression) { Map<String, TreeMap<Resource, ResourceRequestInfo>> remoteRequests = this.remoteRequestsTable.get(priority); if (remoteRequests == null) { remoteRequests = new HashMap<String, TreeMap<Resource, ResourceRequestInfo>>(); this.remoteRequestsTable.put(priority, remoteRequests); if (LOG.isDebugEnabled()) { LOG.debug("Added priority=" + priority); } } TreeMap<Resource, ResourceRequestInfo> reqMap = remoteRequests.get(resourceName); if (reqMap == null) { // capabilities are stored in reverse sorted order. smallest last. reqMap = new TreeMap<Resource, ResourceRequestInfo>(new ResourceReverseMemoryThenCpuComparator()); remoteRequests.put(resourceName, reqMap); } ResourceRequestInfo resourceRequestInfo = reqMap.get(capability); if (resourceRequestInfo == null) { resourceRequestInfo = new ResourceRequestInfo(priority, resourceName, capability, relaxLocality); reqMap.put(capability, resourceRequestInfo); } resourceRequestInfo.remoteRequest.setNumContainers( resourceRequestInfo.remoteRequest.getNumContainers() + 1); if (relaxLocality) { resourceRequestInfo.containerRequests.add(req); } resourceRequestInfo.remoteRequest.setNodeLabelExpression(labelExpression); // Note this down for next interaction with ResourceManager addResourceRequestToAsk(resourceRequestInfo.remoteRequest); if (LOG.isDebugEnabled()) { LOG.debug( "addResourceRequest:" + " applicationId=" + " priority=" + priority.getPriority() + " resourceName=" + resourceName + " numContainers=" + resourceRequestInfo.remoteRequest.getNumContainers() + " #asks=" + ask.size()); } }
private void addResourceRequest(Priority priority, String resourceName, Resource capability) { Map<String, Map<Resource, ResourceRequest>> remoteRequests = this.remoteRequestsTable.get(priority); if (remoteRequests == null) { remoteRequests = new HashMap<String, Map<Resource, ResourceRequest>>(); this.remoteRequestsTable.put(priority, remoteRequests); if (LOG.isDebugEnabled()) { LOG.debug("Added priority=" + priority); } } Map<Resource, ResourceRequest> reqMap = remoteRequests.get(resourceName); if (reqMap == null) { reqMap = new HashMap<Resource, ResourceRequest>(); remoteRequests.put(resourceName, reqMap); } ResourceRequest remoteRequest = reqMap.get(capability); if (remoteRequest == null) { remoteRequest = recordFactory.newRecordInstance(ResourceRequest.class); remoteRequest.setPriority(priority); remoteRequest.setResourceName(resourceName); remoteRequest.setCapability(capability); remoteRequest.setNumContainers(0); reqMap.put(capability, remoteRequest); } remoteRequest.setNumContainers(remoteRequest.getNumContainers() + 1); // Note this down for next interaction with ResourceManager addResourceRequestToAsk(remoteRequest); if (LOG.isDebugEnabled()) { LOG.debug( "addResourceRequest:" + " applicationId=" + applicationId.getId() + " priority=" + priority.getPriority() + " resourceName=" + resourceName + " numContainers=" + remoteRequest.getNumContainers() + " #asks=" + ask.size()); } }
private void decResourceRequest( Priority priority, String resourceName, Resource capability, T req) { Map<String, TreeMap<Resource, ResourceRequestInfo>> remoteRequests = this.remoteRequestsTable.get(priority); if (remoteRequests == null) { if (LOG.isDebugEnabled()) { LOG.debug( "Not decrementing resource as priority " + priority + " is not present in request table"); } return; } Map<Resource, ResourceRequestInfo> reqMap = remoteRequests.get(resourceName); if (reqMap == null) { if (LOG.isDebugEnabled()) { LOG.debug( "Not decrementing resource as " + resourceName + " is not present in request table"); } return; } ResourceRequestInfo resourceRequestInfo = reqMap.get(capability); if (LOG.isDebugEnabled()) { LOG.debug( "BEFORE decResourceRequest:" + " applicationId=" + " priority=" + priority.getPriority() + " resourceName=" + resourceName + " numContainers=" + resourceRequestInfo.remoteRequest.getNumContainers() + " #asks=" + ask.size()); } resourceRequestInfo.remoteRequest.setNumContainers( resourceRequestInfo.remoteRequest.getNumContainers() - 1); resourceRequestInfo.containerRequests.remove(req); if (resourceRequestInfo.remoteRequest.getNumContainers() < 0) { // guard against spurious removals resourceRequestInfo.remoteRequest.setNumContainers(0); } // send the ResourceRequest to RM even if is 0 because it needs to override // a previously sent value. If ResourceRequest was not sent previously then // sending 0 aught to be a no-op on RM addResourceRequestToAsk(resourceRequestInfo.remoteRequest); // delete entries from map if no longer needed if (resourceRequestInfo.remoteRequest.getNumContainers() == 0) { reqMap.remove(capability); if (reqMap.size() == 0) { remoteRequests.remove(resourceName); } if (remoteRequests.size() == 0) { remoteRequestsTable.remove(priority); } } if (LOG.isDebugEnabled()) { LOG.info( "AFTER decResourceRequest:" + " applicationId=" + " priority=" + priority.getPriority() + " resourceName=" + resourceName + " numContainers=" + resourceRequestInfo.remoteRequest.getNumContainers() + " #asks=" + ask.size()); } }
private void decResourceRequest(Priority priority, String resourceName, Resource capability) { Map<String, Map<Resource, ResourceRequest>> remoteRequests = this.remoteRequestsTable.get(priority); Map<Resource, ResourceRequest> reqMap = remoteRequests.get(resourceName); if (reqMap == null) { // as we modify the resource requests by filtering out blacklisted hosts // when they are added, this value may be null when being // decremented if (LOG.isDebugEnabled()) { LOG.debug( "Not decrementing resource as " + resourceName + " is not present in request table"); } return; } ResourceRequest remoteRequest = reqMap.get(capability); if (LOG.isDebugEnabled()) { LOG.debug( "BEFORE decResourceRequest:" + " applicationId=" + applicationId.getId() + " priority=" + priority.getPriority() + " resourceName=" + resourceName + " numContainers=" + remoteRequest.getNumContainers() + " #asks=" + ask.size()); } if (remoteRequest.getNumContainers() > 0) { // based on blacklisting comments above we can end up decrementing more // than requested. so guard for that. remoteRequest.setNumContainers(remoteRequest.getNumContainers() - 1); } if (remoteRequest.getNumContainers() == 0) { reqMap.remove(capability); if (reqMap.size() == 0) { remoteRequests.remove(resourceName); } if (remoteRequests.size() == 0) { remoteRequestsTable.remove(priority); } } // send the updated resource request to RM // send 0 container count requests also to cancel previous requests addResourceRequestToAsk(remoteRequest); if (LOG.isDebugEnabled()) { LOG.info( "AFTER decResourceRequest:" + " applicationId=" + applicationId.getId() + " priority=" + priority.getPriority() + " resourceName=" + resourceName + " numContainers=" + remoteRequest.getNumContainers() + " #asks=" + ask.size()); } }