MV-HarveyNET is a visual dataset of segmented multiple views of the residential buildings damaged due to Hurricane Harvey. This dataset consists of images captured from 5 different views of 435 residential buildings located in Aransas County of Rockport, Texas. These buildings are predominantly single-story, wooden frame residential buildings that were damaged in the 2017 Hurricane Harvey. Each of these buildings are annotated by a team of expert damage inspectors from StEER using damage states labels in accordance with the guidelines in the FEMA’s HAZUS-MH Hurricane model. The expert-assigned damage states to each of these buildings can be used as the ground-truth for model training and testing. The primary goal of compiling this data is to have a dependable training dataset for developing and calibrating image-based multi-view deep learning models for improved automated post-disaster damage assessment.