In this study, we incorporate analyses of genome-wide sequence and structural alterations with pre- and on-therapy transcriptomic and T cell repertoire features in immunotherapy-naive melanoma patients treated with immune checkpoint blockade. Although tumor mutation burden is associated with improved treatment response, the mutation frequency in expressed genes is superior in predicting outcome. Increased T cell density in baseline tumors and dynamic changes in regression or expansion of the T cell repertoire during therapy distinguish responders from non-responders. Transcriptome analyses reveal an increased abundance of B cell subsets in tumors from responders and patterns of molecular response related to expressed mutation elimination or retention that reflect clinical outcome. High-dimensional genomic, transcriptomic, and immune repertoire data were integrated into a multi-modal predictor of response. These findings identify genomic and transcriptomic characteristics of tumors and immune cells that predict response to immune checkpoint blockade and highlight the importance of pre-existing T and B cell immunity in therapeutic outcomes.